Sep 11 13:48:52.133912 ip-10-0-139-201 systemd[1]: Starting Kubernetes Kubelet... Sep 11 13:48:52.516778 ip-10-0-139-201 kubenswrapper[2519]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:48:52.516778 ip-10-0-139-201 kubenswrapper[2519]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 13:48:52.516778 ip-10-0-139-201 kubenswrapper[2519]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:48:52.516778 ip-10-0-139-201 kubenswrapper[2519]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 13:48:52.516778 ip-10-0-139-201 kubenswrapper[2519]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:48:52.517444 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.516840 2519 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 13:48:52.519792 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519776 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.519792 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519791 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519795 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519799 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519802 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519805 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519808 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519811 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519815 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519818 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519820 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519823 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519826 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519828 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519831 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519834 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519836 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519839 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519841 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519845 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.519858 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519847 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519850 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519852 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519855 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519857 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519860 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519863 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519865 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519869 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519872 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519874 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519877 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519880 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519882 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519885 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519887 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519890 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519892 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519895 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519898 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.520305 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519900 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519903 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519906 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519908 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519911 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519913 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519916 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519918 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519921 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519923 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519927 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519929 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519932 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519935 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519938 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519940 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519943 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519946 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519948 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.520847 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519951 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519953 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519958 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519962 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519965 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519968 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519971 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519975 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519978 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519981 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519983 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519986 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519989 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519992 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519994 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.519998 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520000 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520003 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520008 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.521307 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520010 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520014 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520018 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520020 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520024 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520027 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520029 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520032 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520035 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520040 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520044 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520047 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520050 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520052 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520055 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520058 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520061 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520064 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520066 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520069 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.521784 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520072 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520794 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520801 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520805 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520808 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520810 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520813 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520816 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520818 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520821 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520824 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520826 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520830 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520833 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520836 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520839 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520842 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520844 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520847 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520850 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.522265 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520853 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520858 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520862 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520865 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520867 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520870 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520873 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520876 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520878 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520881 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520884 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520887 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520889 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520892 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520894 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520897 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520900 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520902 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520905 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.522793 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520908 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520910 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520913 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520915 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520918 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520922 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520926 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520929 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520931 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520934 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520937 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520939 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520942 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520945 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520948 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520951 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520953 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520956 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520959 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520961 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.523251 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520964 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520966 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520969 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520972 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520974 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520977 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520982 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520985 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520988 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520990 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520993 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520996 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.520999 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521001 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521004 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521007 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521010 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521013 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521016 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521018 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.523788 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521021 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521024 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521027 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521030 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521034 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521037 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521040 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521043 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521045 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521048 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521050 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521053 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521056 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521058 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521061 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521063 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521066 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521068 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521071 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521073 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.524285 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521076 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521457 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521462 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521464 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521468 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521470 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521473 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521476 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521480 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521484 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521487 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521491 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521494 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521496 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521499 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521502 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521505 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521508 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521510 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.524810 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521513 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521516 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521520 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521523 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521527 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521530 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521533 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521536 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521538 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521541 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521543 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521546 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521548 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521551 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521554 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521556 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521559 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521561 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521564 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521566 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.525275 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521569 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521589 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521594 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521598 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521602 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521607 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521612 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521616 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521620 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521624 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521628 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521632 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521635 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521638 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521641 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521644 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521646 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521649 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521651 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521654 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.525782 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521656 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521659 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521661 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521664 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521667 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521670 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521673 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521676 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521678 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521681 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521683 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521686 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521688 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521692 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521694 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521697 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521699 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521702 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521705 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521708 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.526268 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521711 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521716 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521718 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521721 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521723 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521726 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521728 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521731 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521733 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521736 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521738 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521741 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521744 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521746 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521749 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521751 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521754 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521756 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521759 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521761 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.526861 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.521764 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521835 2519 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521873 2519 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521879 2519 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521884 2519 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521889 2519 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521892 2519 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521897 2519 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521901 2519 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521904 2519 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521907 2519 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521912 2519 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521915 2519 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521919 2519 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521922 2519 flags.go:64] FLAG: --cgroup-root="" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521925 2519 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521928 2519 flags.go:64] FLAG: --client-ca-file="" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521931 2519 flags.go:64] FLAG: --cloud-provider="external" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521934 2519 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521940 2519 flags.go:64] FLAG: --cluster-domain="" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521943 2519 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521946 2519 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521949 2519 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521952 2519 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 13:48:52.527352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521956 2519 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521959 2519 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521962 2519 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521965 2519 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521971 2519 flags.go:64] FLAG: --contention-profiling="false" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521974 2519 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521977 2519 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521980 2519 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521983 2519 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521987 2519 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521991 2519 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521993 2519 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.521997 2519 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522000 2519 flags.go:64] FLAG: --enable-server="true" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522003 2519 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522007 2519 flags.go:64] FLAG: --event-burst="100" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522011 2519 flags.go:64] FLAG: --event-qps="50" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522014 2519 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522017 2519 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522020 2519 flags.go:64] FLAG: --eviction-hard="" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522024 2519 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522028 2519 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522031 2519 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522034 2519 flags.go:64] FLAG: --eviction-soft="" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522037 2519 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 13:48:52.527952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522040 2519 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522043 2519 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522046 2519 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522049 2519 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522052 2519 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522055 2519 flags.go:64] FLAG: --feature-gates="" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522058 2519 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522061 2519 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522064 2519 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522067 2519 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522070 2519 flags.go:64] FLAG: --healthz-port="10248" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522075 2519 flags.go:64] FLAG: --help="false" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522078 2519 flags.go:64] FLAG: --hostname-override="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522081 2519 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522084 2519 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522087 2519 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522090 2519 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522094 2519 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522097 2519 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522100 2519 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522103 2519 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522105 2519 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522108 2519 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522111 2519 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 13:48:52.528662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522114 2519 flags.go:64] FLAG: --kube-reserved="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522117 2519 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522120 2519 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522123 2519 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522130 2519 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522133 2519 flags.go:64] FLAG: --lock-file="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522136 2519 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522139 2519 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522142 2519 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522147 2519 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522150 2519 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522153 2519 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522155 2519 flags.go:64] FLAG: --logging-format="text" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522158 2519 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522162 2519 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522165 2519 flags.go:64] FLAG: --manifest-url="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522167 2519 flags.go:64] FLAG: --manifest-url-header="" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522172 2519 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522175 2519 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522181 2519 flags.go:64] FLAG: --max-pods="110" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522184 2519 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522187 2519 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522190 2519 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522193 2519 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522196 2519 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 13:48:52.529259 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522199 2519 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522202 2519 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522210 2519 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522213 2519 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522216 2519 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522219 2519 flags.go:64] FLAG: --pod-cidr="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522222 2519 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522227 2519 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522230 2519 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522233 2519 flags.go:64] FLAG: --pods-per-core="0" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522236 2519 flags.go:64] FLAG: --port="10250" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522239 2519 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522242 2519 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-055459aa43120abdd" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522246 2519 flags.go:64] FLAG: --qos-reserved="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522249 2519 flags.go:64] FLAG: --read-only-port="10255" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522252 2519 flags.go:64] FLAG: --register-node="true" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522256 2519 flags.go:64] FLAG: --register-with-taints="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522260 2519 flags.go:64] FLAG: --registry-burst="10" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522262 2519 flags.go:64] FLAG: --registry-qps="5" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522265 2519 flags.go:64] FLAG: --reserved-cpus="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522268 2519 flags.go:64] FLAG: --reserved-memory="" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522272 2519 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522275 2519 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522278 2519 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522281 2519 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 13:48:52.529892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522284 2519 flags.go:64] FLAG: --runonce="false" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522287 2519 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522291 2519 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522294 2519 flags.go:64] FLAG: --seccomp-default="false" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522298 2519 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522301 2519 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522305 2519 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522308 2519 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522311 2519 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522314 2519 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522317 2519 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522320 2519 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522322 2519 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522325 2519 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522328 2519 flags.go:64] FLAG: --system-cgroups="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522331 2519 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522336 2519 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522339 2519 flags.go:64] FLAG: --tls-cert-file="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522342 2519 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522347 2519 flags.go:64] FLAG: --tls-min-version="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522350 2519 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522353 2519 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522356 2519 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522359 2519 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522361 2519 flags.go:64] FLAG: --v="2" Sep 11 13:48:52.530490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522365 2519 flags.go:64] FLAG: --version="false" Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522369 2519 flags.go:64] FLAG: --vmodule="" Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522373 2519 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.522376 2519 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522490 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522494 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522497 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522500 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522503 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522507 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522510 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522514 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522517 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522520 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522522 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522525 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522527 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522530 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522533 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522535 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.531138 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522538 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522541 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522543 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522546 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522550 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522554 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522557 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522560 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522563 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522566 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522569 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522586 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522589 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522592 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522594 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522597 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522600 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522603 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522605 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.531653 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522608 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522611 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522615 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522618 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522622 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522625 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522628 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522631 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522633 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522636 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522638 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522641 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522644 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522646 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522649 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522652 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522654 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522657 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522660 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522663 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.532118 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522665 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522668 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522671 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522674 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522676 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522679 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522681 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522684 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522687 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522689 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522692 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522694 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522697 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522700 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522704 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522706 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522717 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522721 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522724 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522727 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.532673 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522730 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522733 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522736 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522738 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522742 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522746 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522749 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522752 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522755 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522759 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522763 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522766 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522769 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522772 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522774 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522778 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522781 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522784 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522787 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.533201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522790 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522792 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522795 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522798 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.522801 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.523291 2519 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530124 2519 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530139 2519 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530159 2519 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530164 2519 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530230 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530234 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530238 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530242 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530245 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.533684 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530248 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530251 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530254 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530256 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530259 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530261 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530264 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530267 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530269 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530272 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530275 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530278 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530280 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530284 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530287 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530289 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530292 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530295 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530297 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530300 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.534062 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530303 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530305 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530308 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530312 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530317 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530320 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530323 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530326 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530328 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530331 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530335 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530339 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530342 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530344 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530347 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530350 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530352 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530355 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530357 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.534542 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530360 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530363 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530365 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530367 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530370 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530372 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530375 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530378 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530381 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530384 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530387 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530390 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530392 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530394 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530397 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530399 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530402 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530405 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530408 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530411 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530413 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.535038 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530416 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530418 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530421 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530423 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530426 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530428 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530431 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530434 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530437 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530439 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530442 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530444 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530447 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530449 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530452 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530454 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530457 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530459 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530462 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530465 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.535543 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530467 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530470 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530472 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530475 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530477 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530480 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530483 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530486 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530490 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530493 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530496 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530499 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530502 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530505 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.536113 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530510 2519 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530633 2519 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530639 2519 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530643 2519 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530646 2519 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530649 2519 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530652 2519 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530655 2519 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530660 2519 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530663 2519 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530665 2519 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530668 2519 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530671 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530674 2519 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530677 2519 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530679 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530681 2519 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530684 2519 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530687 2519 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530690 2519 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:48:52.536501 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530692 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530695 2519 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530697 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530700 2519 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530702 2519 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530705 2519 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530708 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530717 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530720 2519 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530723 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530725 2519 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530728 2519 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530730 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530734 2519 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530736 2519 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530739 2519 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530742 2519 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530745 2519 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530748 2519 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:48:52.537017 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530750 2519 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530753 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530756 2519 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530758 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530761 2519 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530763 2519 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530766 2519 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530769 2519 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530771 2519 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530774 2519 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530776 2519 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530779 2519 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530783 2519 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530786 2519 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530789 2519 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530792 2519 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530794 2519 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530797 2519 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530800 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530802 2519 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:48:52.537480 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530805 2519 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530807 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530810 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530812 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530815 2519 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530817 2519 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530820 2519 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530823 2519 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530825 2519 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530828 2519 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530830 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530833 2519 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530835 2519 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530838 2519 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530840 2519 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530843 2519 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530846 2519 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530849 2519 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530852 2519 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:48:52.537979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530854 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530857 2519 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530859 2519 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530862 2519 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530866 2519 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530869 2519 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530872 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530875 2519 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530878 2519 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530880 2519 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530883 2519 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530886 2519 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530888 2519 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530891 2519 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530893 2519 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530896 2519 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530898 2519 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530901 2519 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530903 2519 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530906 2519 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:48:52.538433 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530909 2519 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:52.530911 2519 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.530916 2519 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.531012 2519 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.533716 2519 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.534616 2519 server.go:1019] "Starting client certificate rotation" Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.534724 2519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.534765 2519 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:48:52.538958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.536970 2519 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:48:52.542770 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.542746 2519 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:48:52.549923 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.549907 2519 log.go:25] "Validated CRI v1 runtime API" Sep 11 13:48:52.553057 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.553039 2519 log.go:25] "Validated CRI v1 image API" Sep 11 13:48:52.554815 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.554799 2519 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 13:48:52.557781 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.557758 2519 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 943d1292-a9be-4a1a-befa-ecdbf822f0ac:/dev/nvme0n1p4 c939333a-9e35-413c-ada1-971fdabe5535:/dev/nvme0n1p3] Sep 11 13:48:52.557841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.557781 2519 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 13:48:52.561099 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.561076 2519 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:48:52.563411 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.563302 2519 manager.go:216] Machine: {Timestamp:2026-09-11 13:48:52.561933974 +0000 UTC m=+0.316248857 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101928 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f490a5b0f5214d62beb4f7f26f69 SystemUUID:ec25f490-a5b0-f521-4d62-beb4f7f26f69 BootID:857f1654-f328-4674-90f9-df6ec17076b0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:45:02:3a:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:45:02:3a:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:67:9c:56:da:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 13:48:52.563411 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.563408 2519 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 13:48:52.563518 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.563487 2519 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 13:48:52.566427 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.566403 2519 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 13:48:52.566553 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.566428 2519 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 13:48:52.567075 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.567064 2519 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 13:48:52.567112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.567076 2519 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 13:48:52.567112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.567085 2519 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:48:52.567598 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.567588 2519 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:48:52.568327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.568318 2519 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 13:48:52.571829 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.571817 2519 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:48:52.571948 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.571937 2519 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 13:48:52.571987 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.571969 2519 kubelet.go:486] "Attempting to sync node with API server" Sep 11 13:48:52.571987 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.571978 2519 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 13:48:52.572051 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.571992 2519 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 13:48:52.572051 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.572001 2519 kubelet.go:398] "Adding apiserver pod source" Sep 11 13:48:52.572051 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.572010 2519 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 13:48:52.572375 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.572363 2519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:48:52.572424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.572385 2519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:48:52.574706 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.574691 2519 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 13:48:52.576755 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.576740 2519 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 13:48:52.576820 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.576772 2519 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 13:48:52.578200 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578184 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 13:48:52.578200 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578207 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578214 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578221 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578230 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578237 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578251 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578258 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578269 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578277 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578286 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 13:48:52.578327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.578292 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 13:48:52.579140 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.579131 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 13:48:52.579140 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.579141 2519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 13:48:52.582513 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.582493 2519 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 13:48:52.582513 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.582497 2519 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 13:48:52.582669 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.582537 2519 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 13:48:52.582856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.582842 2519 server.go:1268] "Started kubelet" Sep 11 13:48:52.583607 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.583548 2519 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 13:48:52.583693 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.583607 2519 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 13:48:52.583693 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.583672 2519 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 13:48:52.584678 ip-10-0-139-201 systemd[1]: Started Kubernetes Kubelet. Sep 11 13:48:52.586541 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.586522 2519 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 13:48:52.587833 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.587818 2519 server.go:310] "Adding debug handlers to kubelet server" Sep 11 13:48:52.588027 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.587883 2519 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k62jz" Sep 11 13:48:52.589948 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.589922 2519 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 13:48:52.592872 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.591938 2519 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-201.ec2.internal.18d44860cdfbbd4e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-201.ec2.internal,UID:ip-10-0-139-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-201.ec2.internal,},FirstTimestamp:2026-09-11 13:48:52.582817102 +0000 UTC m=+0.337132047,LastTimestamp:2026-09-11 13:48:52.582817102 +0000 UTC m=+0.337132047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-201.ec2.internal,}" Sep 11 13:48:52.593305 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.593285 2519 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k62jz" Sep 11 13:48:52.594402 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.594384 2519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 13:48:52.595016 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595001 2519 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 13:48:52.595145 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.595119 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:52.595615 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595598 2519 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 13:48:52.595696 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595634 2519 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 13:48:52.595696 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595648 2519 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 13:48:52.595810 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595728 2519 reconstruct.go:97] "Volume reconstruction finished" Sep 11 13:48:52.595810 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595739 2519 reconciler.go:29] "Reconciler: start to sync state" Sep 11 13:48:52.595925 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595911 2519 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 13:48:52.595925 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595925 2519 factory.go:55] Registering systemd factory Sep 11 13:48:52.596010 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.595931 2519 factory.go:223] Registration of the systemd container factory successfully Sep 11 13:48:52.596123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.596110 2519 factory.go:153] Registering CRI-O factory Sep 11 13:48:52.596123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.596122 2519 factory.go:223] Registration of the crio container factory successfully Sep 11 13:48:52.596223 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.596135 2519 factory.go:103] Registering Raw factory Sep 11 13:48:52.596223 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.596144 2519 manager.go:1196] Started watching for new ooms in manager Sep 11 13:48:52.596508 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.596495 2519 manager.go:349] Starting recovery of all containers Sep 11 13:48:52.607070 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.607052 2519 manager.go:354] Recovery completed Sep 11 13:48:52.607292 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.607273 2519 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:52.610154 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.609963 2519 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.611759 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.611747 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.614817 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.614796 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.614907 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.614826 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.614907 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.614837 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.615626 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.615608 2519 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 13:48:52.615701 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.615627 2519 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 13:48:52.615701 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.615647 2519 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:48:52.615866 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.615847 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.617781 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.617769 2519 policy_none.go:49] "None policy: Start" Sep 11 13:48:52.617847 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.617785 2519 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 13:48:52.617847 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.617796 2519 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 13:48:52.619803 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.619791 2519 policy_none.go:47] "Start" Sep 11 13:48:52.620601 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.620568 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.620676 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.620613 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.620676 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.620624 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661171 2519 manager.go:337] "Starting Device Plugin manager" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.661205 2519 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661217 2519 server.go:87] "Starting device plugin registration server" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661591 2519 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661605 2519 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661714 2519 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661810 2519 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.661820 2519 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.662374 2519 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 13:48:52.669645 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.662426 2519 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:52.682895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.682866 2519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 13:48:52.684140 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.684125 2519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 13:48:52.684204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.684171 2519 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 13:48:52.684204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.684197 2519 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 13:48:52.684308 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.684291 2519 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 13:48:52.686567 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.686546 2519 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:52.761762 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.761739 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.762794 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.762780 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.762851 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.762808 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.762851 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.762819 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.762851 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.762850 2519 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.770650 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.770606 2519 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.770650 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.770626 2519 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-201.ec2.internal\": node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:52.784793 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.784771 2519 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal"] Sep 11 13:48:52.784890 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.784840 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.785091 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.785077 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:52.786269 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.786243 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.786380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.786280 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.786380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.786297 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.788697 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.788685 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.788826 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.788810 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.788889 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.788855 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.789455 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789439 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.789455 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789447 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.789589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789467 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.789589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789482 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.789589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789469 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.789589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.789561 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.791922 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.791906 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.791991 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.791931 2519 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:48:52.792725 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.792709 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:48:52.792790 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.792737 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:48:52.792790 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.792753 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:48:52.796699 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.796679 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.796782 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.796707 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.796782 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.796724 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.810255 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.810238 2519 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.814739 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.814721 2519 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-201.ec2.internal\" not found" node="ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.885794 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.885763 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:52.897616 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897592 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.897748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897622 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.897748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897642 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.897748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897673 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.897748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897680 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b23ec63d9b3d220644ea926d481ee00f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-201.ec2.internal\" (UID: \"b23ec63d9b3d220644ea926d481ee00f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.897748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:52.897727 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cef12fc99af254fa0de9a3b0e067caf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal\" (UID: \"cef12fc99af254fa0de9a3b0e067caf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:52.986927 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:52.986889 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:53.087702 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.087622 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:53.113072 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.113043 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:53.118782 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.117985 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:53.188484 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.188440 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:53.293932 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.293906 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:53.394984 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.394905 2519 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-201.ec2.internal\" not found" Sep 11 13:48:53.484183 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.484159 2519 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:53.495572 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.495533 2519 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" Sep 11 13:48:53.506659 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.506637 2519 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 13:48:53.508513 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.508500 2519 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" Sep 11 13:48:53.519563 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.519544 2519 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 13:48:53.534738 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.534711 2519 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 13:48:53.534869 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.534852 2519 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:48:53.534914 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.534891 2519 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:48:53.534914 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.534901 2519 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:48:53.574042 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.574012 2519 apiserver.go:52] "Watching apiserver" Sep 11 13:48:53.586753 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.586725 2519 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 13:48:53.590239 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.589687 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rkqt7","kube-system/konnectivity-agent-8lngk","kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb","openshift-image-registry/node-ca-lvrq9","openshift-multus/network-metrics-daemon-p8bbp","openshift-network-operator/iptables-alerter-zpk9r","openshift-cluster-node-tuning-operator/tuned-mwdc5","openshift-dns/node-resolver-pdvmj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal","openshift-multus/multus-additional-cni-plugins-8th59","openshift-multus/multus-vc85z","openshift-network-diagnostics/network-check-target-h6n6n","openshift-ovn-kubernetes/ovnkube-node-bqvz8"] Sep 11 13:48:53.592711 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.592676 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.594965 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.594944 2519 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 13:48:53.595119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.595097 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hfrnc\"" Sep 11 13:48:53.595119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.595109 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.595258 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.595125 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.595314 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.595114 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.596075 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.596037 2519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:43:52 +0000 UTC" deadline="2028-05-07 02:32:01.845996603 +0000 UTC" Sep 11 13:48:53.596075 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.596073 2519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14484h43m8.249927217s" Sep 11 13:48:53.596940 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.596917 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nrtzh\"" Sep 11 13:48:53.597035 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.596973 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.597137 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.597122 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 13:48:53.597549 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.597529 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.597664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.597529 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 13:48:53.597664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.597564 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 13:48:53.597664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.597659 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 13:48:53.598160 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.598142 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:53.598255 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.598205 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:48:53.600395 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600377 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.600776 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600756 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b4a9107-64d8-45be-bfe9-ed28093f3c83-hosts-file\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.600843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600783 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4a9107-64d8-45be-bfe9-ed28093f3c83-tmp-dir\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.600843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600799 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-kubelet\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.600843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600814 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-systemd-units\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.600843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600837 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-ovn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600863 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600905 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-slash\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600929 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600961 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600979 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-script-lib\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.600996 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:53.601048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601014 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfzk\" (UniqueName: \"kubernetes.io/projected/3b4a9107-64d8-45be-bfe9-ed28093f3c83-kube-api-access-nhfzk\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601052 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-netns\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601068 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-systemd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601098 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-netd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601153 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-config\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601173 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-env-overrides\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601189 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-var-lib-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601208 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-etc-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601232 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-node-log\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601252 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-log-socket\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601699 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601294 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-bin\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601699 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601318 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36998c19-023d-4d86-8313-676fd0d6527b-ovn-node-metrics-cert\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.601699 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.601335 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgrn\" (UniqueName: \"kubernetes.io/projected/36998c19-023d-4d86-8313-676fd0d6527b-kube-api-access-8vgrn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.602086 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.602068 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 13:48:53.602149 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.602127 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78m7q\"" Sep 11 13:48:53.602240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.602224 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.602451 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.602439 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.602843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.602825 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.604781 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.604763 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 13:48:53.604856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.604841 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.604898 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.604879 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.605098 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.605080 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 13:48:53.605172 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.605133 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hrtrm\"" Sep 11 13:48:53.605343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.605329 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.605400 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.605388 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:48:53.605984 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.605965 2519 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:53.606983 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.606966 2519 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:48:53.607861 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.607845 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.609844 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.609826 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.610097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.610082 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x4c9z\"" Sep 11 13:48:53.610177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.610122 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 13:48:53.610503 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.610489 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.611042 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.611022 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.613180 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.612696 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.613180 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.612968 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.613180 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.613094 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ppkxx\"" Sep 11 13:48:53.613180 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.613139 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.614372 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.613256 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:48:53.616558 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.616537 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.619073 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.619051 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.621664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.621629 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.625813 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.625770 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 13:48:53.625964 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.625847 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 13:48:53.625964 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.625867 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-df6pg\"" Sep 11 13:48:53.625964 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.625866 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 13:48:53.626103 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.625973 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 13:48:53.626103 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.626014 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-schgd\"" Sep 11 13:48:53.626204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.626152 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9lplj\"" Sep 11 13:48:53.626258 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.626151 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 13:48:53.626313 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.626302 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 13:48:53.626368 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.626356 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 13:48:53.631231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.631213 2519 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx7sh" Sep 11 13:48:53.639845 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.639829 2519 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx7sh" Sep 11 13:48:53.696599 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.696576 2519 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 13:48:53.701913 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.701888 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-var-lib-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702004 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.701927 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-etc-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702004 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.701953 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-log-socket\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702004 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.701978 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36998c19-023d-4d86-8313-676fd0d6527b-ovn-node-metrics-cert\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702004 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.701994 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-etc-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702002 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgrn\" (UniqueName: \"kubernetes.io/projected/36998c19-023d-4d86-8313-676fd0d6527b-kube-api-access-8vgrn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702058 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-log-socket\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702061 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysconfig\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702078 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-var-lib-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702130 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-socket-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702155 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4a9107-64d8-45be-bfe9-ed28093f3c83-tmp-dir\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.702176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702173 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-dbus\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702196 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702226 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wlx\" (UniqueName: \"kubernetes.io/projected/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-kube-api-access-p6wlx\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702280 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-sys-fs\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702335 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702364 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-tuned\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702389 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-bin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702403 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-openvswitch\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702416 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cnibin\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702443 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-netns\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702444 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b4a9107-64d8-45be-bfe9-ed28093f3c83-tmp-dir\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702475 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-systemd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702501 2519 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702521 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-netns\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702505 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-config\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.702564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702543 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-systemd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702572 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-etc-kubernetes\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702622 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702654 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-modprobe-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702685 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-tmp\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702711 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-system-cni-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702729 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702744 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702766 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5be8ff13-5b2e-409c-bebc-84a07e44ef77-iptables-alerter-script\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702803 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5be8ff13-5b2e-409c-bebc-84a07e44ef77-host-slash\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702834 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-kubelet\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702859 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702884 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-host\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702889 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-kubelet\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702907 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-netns\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702919 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702933 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-hostroot\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.703304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702960 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-multus-certs\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.702985 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-kubelet-config\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703009 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-slash\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703032 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84a9e05-9178-4368-8744-1b49ee4ba041-serviceca\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703059 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5pl\" (UniqueName: \"kubernetes.io/projected/b84a9e05-9178-4368-8744-1b49ee4ba041-kube-api-access-rx5pl\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703080 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703102 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703110 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-slash\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703120 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-config\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703131 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwm2g\" (UniqueName: \"kubernetes.io/projected/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-kube-api-access-wwm2g\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703171 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfzk\" (UniqueName: \"kubernetes.io/projected/3b4a9107-64d8-45be-bfe9-ed28093f3c83-kube-api-access-nhfzk\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703216 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-multus\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703242 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703261 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-device-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703313 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs57g\" (UniqueName: \"kubernetes.io/projected/5be8ff13-5b2e-409c-bebc-84a07e44ef77-kube-api-access-rs57g\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703361 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-var-lib-kubelet\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704102 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703388 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-k8s-cni-cncf-io\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703418 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-bin\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703446 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-kubernetes\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703471 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b4a9107-64d8-45be-bfe9-ed28093f3c83-hosts-file\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703499 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-systemd-units\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703554 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-systemd-units\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703556 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b4a9107-64d8-45be-bfe9-ed28093f3c83-hosts-file\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703556 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-bin\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703615 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-ovn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703659 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-run-ovn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.703763 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704053 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704086 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-sys\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704106 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-socket-dir-parent\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704122 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-conf\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704136 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-lib-modules\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704152 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kk6j\" (UniqueName: \"kubernetes.io/projected/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-kube-api-access-6kk6j\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.704695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704169 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbfn\" (UniqueName: \"kubernetes.io/projected/7095be69-101f-4123-ba41-aeaceb4eb62a-kube-api-access-5cbfn\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704208 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-netd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704243 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-env-overrides\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704282 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-host-cni-netd\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704304 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704319 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzcn\" (UniqueName: \"kubernetes.io/projected/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-kube-api-access-lkzcn\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704355 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84a9e05-9178-4368-8744-1b49ee4ba041-host\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704398 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/204076d3-c263-49ce-9747-eea120329029-konnectivity-ca\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704426 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-registration-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704454 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-node-log\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704480 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-systemd\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704506 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-system-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704513 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36998c19-023d-4d86-8313-676fd0d6527b-node-log\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704555 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704585 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704650 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cnibin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704676 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-conf-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704690 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-env-overrides\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704701 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/204076d3-c263-49ce-9747-eea120329029-agent-certs\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704733 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-script-lib\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704759 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704781 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-run\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704808 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cni-binary-copy\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704835 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-daemon-config\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.704880 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-kubelet\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.705894 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.705159 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36998c19-023d-4d86-8313-676fd0d6527b-ovnkube-script-lib\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.706494 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.706472 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36998c19-023d-4d86-8313-676fd0d6527b-ovn-node-metrics-cert\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.710743 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.710716 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:48:53.710855 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.710748 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:48:53.710855 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.710765 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:53.710855 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.710845 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:48:54.210821509 +0000 UTC m=+1.965136397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:53.713886 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.712137 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgrn\" (UniqueName: \"kubernetes.io/projected/36998c19-023d-4d86-8313-676fd0d6527b-kube-api-access-8vgrn\") pod \"ovnkube-node-bqvz8\" (UID: \"36998c19-023d-4d86-8313-676fd0d6527b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.714526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.714193 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfzk\" (UniqueName: \"kubernetes.io/projected/3b4a9107-64d8-45be-bfe9-ed28093f3c83-kube-api-access-nhfzk\") pod \"node-resolver-pdvmj\" (UID: \"3b4a9107-64d8-45be-bfe9-ed28093f3c83\") " pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.805231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805198 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-tuned\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.805231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805233 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-bin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805252 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cnibin\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805277 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-etc-kubernetes\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805320 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-bin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805327 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-etc-kubernetes\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805323 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805363 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-modprobe-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805388 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-tmp\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.805423 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:53.805435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805431 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-system-cni-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805460 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805363 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cnibin\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805478 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-modprobe-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.805488 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:48:54.305467382 +0000 UTC m=+2.059782255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805506 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-system-cni-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805626 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805659 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5be8ff13-5b2e-409c-bebc-84a07e44ef77-iptables-alerter-script\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805695 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5be8ff13-5b2e-409c-bebc-84a07e44ef77-host-slash\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805723 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-host\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805747 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-netns\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805769 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-hostroot\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805795 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-multus-certs\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805801 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5be8ff13-5b2e-409c-bebc-84a07e44ef77-host-slash\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805823 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-kubelet-config\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805847 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-netns\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805855 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84a9e05-9178-4368-8744-1b49ee4ba041-serviceca\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.805912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805859 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-hostroot\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805806 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-host\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805889 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5pl\" (UniqueName: \"kubernetes.io/projected/b84a9e05-9178-4368-8744-1b49ee4ba041-kube-api-access-rx5pl\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805921 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-multus-certs\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805916 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805970 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805988 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-kubelet-config\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.805997 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwm2g\" (UniqueName: \"kubernetes.io/projected/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-kube-api-access-wwm2g\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806025 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-multus\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806034 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806050 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806078 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-device-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806104 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs57g\" (UniqueName: \"kubernetes.io/projected/5be8ff13-5b2e-409c-bebc-84a07e44ef77-kube-api-access-rs57g\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806147 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-var-lib-kubelet\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806173 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-k8s-cni-cncf-io\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806203 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-kubernetes\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806198 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-binary-copy\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.806823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806229 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5be8ff13-5b2e-409c-bebc-84a07e44ef77-iptables-alerter-script\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806289 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-device-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806297 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-sys\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806233 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-sys\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806320 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806320 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-cni-multus\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806343 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806353 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84a9e05-9178-4368-8744-1b49ee4ba041-serviceca\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806343 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-socket-dir-parent\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806383 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-run-k8s-cni-cncf-io\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806383 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-var-lib-kubelet\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806430 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-socket-dir-parent\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806455 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-kubernetes\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806494 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-conf\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806522 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-lib-modules\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806543 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kk6j\" (UniqueName: \"kubernetes.io/projected/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-kube-api-access-6kk6j\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806608 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbfn\" (UniqueName: \"kubernetes.io/projected/7095be69-101f-4123-ba41-aeaceb4eb62a-kube-api-access-5cbfn\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.807695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806647 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806699 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzcn\" (UniqueName: \"kubernetes.io/projected/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-kube-api-access-lkzcn\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806711 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-conf\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806723 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84a9e05-9178-4368-8744-1b49ee4ba041-host\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806764 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/204076d3-c263-49ce-9747-eea120329029-konnectivity-ca\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806789 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-registration-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806792 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806816 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-systemd\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806839 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-system-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806866 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806896 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806907 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-lib-modules\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806921 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cnibin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806943 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-systemd\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806958 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84a9e05-9178-4368-8744-1b49ee4ba041-host\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806968 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-conf-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.806995 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/204076d3-c263-49ce-9747-eea120329029-agent-certs\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807011 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-registration-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.808591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807035 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-run\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807042 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-system-cni-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807061 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cni-binary-copy\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807061 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-conf-dir\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807089 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-daemon-config\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807131 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cnibin\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807153 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-kubelet\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807187 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysconfig\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807213 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-socket-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807238 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-dbus\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807262 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807288 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wlx\" (UniqueName: \"kubernetes.io/projected/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-kube-api-access-p6wlx\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807312 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-sys-fs\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807404 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-sys-fs\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807557 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/204076d3-c263-49ce-9747-eea120329029-konnectivity-ca\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807580 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807708 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysctl-d\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.809526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807749 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7095be69-101f-4123-ba41-aeaceb4eb62a-socket-dir\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807811 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-run\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807819 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-sysconfig\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807850 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-host-var-lib-kubelet\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.807904 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807910 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21edc280-a387-48e2-a543-d24f2c5152cd-dbus\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.807901 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-multus-daemon-config\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:53.807953 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:48:54.307937746 +0000 UTC m=+2.062252626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.808155 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.808225 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-etc-tuned\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.808611 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-cni-binary-copy\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.808618 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-tmp\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.810071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.809819 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/204076d3-c263-49ce-9747-eea120329029-agent-certs\") pod \"konnectivity-agent-8lngk\" (UID: \"204076d3-c263-49ce-9747-eea120329029\") " pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:53.814439 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.814418 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5pl\" (UniqueName: \"kubernetes.io/projected/b84a9e05-9178-4368-8744-1b49ee4ba041-kube-api-access-rx5pl\") pod \"node-ca-lvrq9\" (UID: \"b84a9e05-9178-4368-8744-1b49ee4ba041\") " pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:53.818053 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.818005 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kk6j\" (UniqueName: \"kubernetes.io/projected/9e8b0ab9-2837-4ab7-875b-9ad03ddd6153-kube-api-access-6kk6j\") pod \"tuned-mwdc5\" (UID: \"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153\") " pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:53.818156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.818135 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzcn\" (UniqueName: \"kubernetes.io/projected/02e73bd8-0cd9-421c-84af-60ca5a3f3ae8-kube-api-access-lkzcn\") pod \"multus-vc85z\" (UID: \"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8\") " pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.818468 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.818450 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs57g\" (UniqueName: \"kubernetes.io/projected/5be8ff13-5b2e-409c-bebc-84a07e44ef77-kube-api-access-rs57g\") pod \"iptables-alerter-zpk9r\" (UID: \"5be8ff13-5b2e-409c-bebc-84a07e44ef77\") " pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:53.818532 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.818511 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbfn\" (UniqueName: \"kubernetes.io/projected/7095be69-101f-4123-ba41-aeaceb4eb62a-kube-api-access-5cbfn\") pod \"aws-ebs-csi-driver-node-6smmb\" (UID: \"7095be69-101f-4123-ba41-aeaceb4eb62a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.818578 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.818537 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwm2g\" (UniqueName: \"kubernetes.io/projected/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-kube-api-access-wwm2g\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:53.819743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.819724 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wlx\" (UniqueName: \"kubernetes.io/projected/5294fa29-44c4-427a-b6c4-5d7ef844a3a3-kube-api-access-p6wlx\") pod \"multus-additional-cni-plugins-8th59\" (UID: \"5294fa29-44c4-427a-b6c4-5d7ef844a3a3\") " pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:53.832478 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.832443 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef12fc99af254fa0de9a3b0e067caf7.slice/crio-c55b376f1db47b29fb927eba972c61ba8e386d4b6ec4f47ae8efab6fe3e1c4e8 WatchSource:0}: Error finding container c55b376f1db47b29fb927eba972c61ba8e386d4b6ec4f47ae8efab6fe3e1c4e8: Status 404 returned error can't find the container with id c55b376f1db47b29fb927eba972c61ba8e386d4b6ec4f47ae8efab6fe3e1c4e8 Sep 11 13:48:53.832912 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.832896 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23ec63d9b3d220644ea926d481ee00f.slice/crio-f003fa3f6d2b6366bfaa9bd0d397469f3c7448c46c556df1a700e57a26b3f077 WatchSource:0}: Error finding container f003fa3f6d2b6366bfaa9bd0d397469f3c7448c46c556df1a700e57a26b3f077: Status 404 returned error can't find the container with id f003fa3f6d2b6366bfaa9bd0d397469f3c7448c46c556df1a700e57a26b3f077 Sep 11 13:48:53.838549 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.838533 2519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 13:48:53.918811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.918710 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdvmj" Sep 11 13:48:53.929461 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.929437 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a9107_64d8_45be_bfe9_ed28093f3c83.slice/crio-dbd320feb12cdbbbd2e706990dc6e2352c7e75df8a87b7a09b5ef0337a744034 WatchSource:0}: Error finding container dbd320feb12cdbbbd2e706990dc6e2352c7e75df8a87b7a09b5ef0337a744034: Status 404 returned error can't find the container with id dbd320feb12cdbbbd2e706990dc6e2352c7e75df8a87b7a09b5ef0337a744034 Sep 11 13:48:53.947173 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.947148 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:48:53.955459 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.955441 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" Sep 11 13:48:53.955747 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.955721 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36998c19_023d_4d86_8313_676fd0d6527b.slice/crio-b05ea7a848e1dce856ba7bd7472530f7c2a5dbea3446a542fd31bf658a633880 WatchSource:0}: Error finding container b05ea7a848e1dce856ba7bd7472530f7c2a5dbea3446a542fd31bf658a633880: Status 404 returned error can't find the container with id b05ea7a848e1dce856ba7bd7472530f7c2a5dbea3446a542fd31bf658a633880 Sep 11 13:48:53.964025 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.963998 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7095be69_101f_4123_ba41_aeaceb4eb62a.slice/crio-ff52c4238b4401d69be5dd8717b9e0cd5d90c73f3d9efb3582a503998579fdcb WatchSource:0}: Error finding container ff52c4238b4401d69be5dd8717b9e0cd5d90c73f3d9efb3582a503998579fdcb: Status 404 returned error can't find the container with id ff52c4238b4401d69be5dd8717b9e0cd5d90c73f3d9efb3582a503998579fdcb Sep 11 13:48:53.980082 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.980055 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc85z" Sep 11 13:48:53.988754 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:53.988730 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e73bd8_0cd9_421c_84af_60ca5a3f3ae8.slice/crio-f8d61b788085cfbafe8658d86ced0396462a2d8731fc426a859762228d4135c3 WatchSource:0}: Error finding container f8d61b788085cfbafe8658d86ced0396462a2d8731fc426a859762228d4135c3: Status 404 returned error can't find the container with id f8d61b788085cfbafe8658d86ced0396462a2d8731fc426a859762228d4135c3 Sep 11 13:48:53.995796 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:53.995780 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpk9r" Sep 11 13:48:54.004956 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:54.004709 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be8ff13_5b2e_409c_bebc_84a07e44ef77.slice/crio-57a65bd9f45c7e44c24916a8d64c0b3f997ee6e45992837d2fcc701fb4e120ae WatchSource:0}: Error finding container 57a65bd9f45c7e44c24916a8d64c0b3f997ee6e45992837d2fcc701fb4e120ae: Status 404 returned error can't find the container with id 57a65bd9f45c7e44c24916a8d64c0b3f997ee6e45992837d2fcc701fb4e120ae Sep 11 13:48:54.010852 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.010830 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" Sep 11 13:48:54.018979 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:54.018950 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8b0ab9_2837_4ab7_875b_9ad03ddd6153.slice/crio-c37501855305eca0aa56f0f1c3b7a2d2ed9b486127406031255f4724c94a0e49 WatchSource:0}: Error finding container c37501855305eca0aa56f0f1c3b7a2d2ed9b486127406031255f4724c94a0e49: Status 404 returned error can't find the container with id c37501855305eca0aa56f0f1c3b7a2d2ed9b486127406031255f4724c94a0e49 Sep 11 13:48:54.024069 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.024051 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8th59" Sep 11 13:48:54.032522 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:54.032500 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5294fa29_44c4_427a_b6c4_5d7ef844a3a3.slice/crio-3fbc590f0432a717a769a68fdf5b09559d43ddfa019c5eb042b46e401bfde176 WatchSource:0}: Error finding container 3fbc590f0432a717a769a68fdf5b09559d43ddfa019c5eb042b46e401bfde176: Status 404 returned error can't find the container with id 3fbc590f0432a717a769a68fdf5b09559d43ddfa019c5eb042b46e401bfde176 Sep 11 13:48:54.045564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.045534 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvrq9" Sep 11 13:48:54.050701 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.050680 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:48:54.054308 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:54.054288 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84a9e05_9178_4368_8744_1b49ee4ba041.slice/crio-44e396c518b22e7f052c40042ed081d854f99d0d8fe40c00cb86ff7c82a0ec4a WatchSource:0}: Error finding container 44e396c518b22e7f052c40042ed081d854f99d0d8fe40c00cb86ff7c82a0ec4a: Status 404 returned error can't find the container with id 44e396c518b22e7f052c40042ed081d854f99d0d8fe40c00cb86ff7c82a0ec4a Sep 11 13:48:54.059532 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:48:54.059511 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204076d3_c263_49ce_9747_eea120329029.slice/crio-5e10964d6b60dd0eaf7d863f277bcef016d4328b8f356089c41321f23d44d2de WatchSource:0}: Error finding container 5e10964d6b60dd0eaf7d863f277bcef016d4328b8f356089c41321f23d44d2de: Status 404 returned error can't find the container with id 5e10964d6b60dd0eaf7d863f277bcef016d4328b8f356089c41321f23d44d2de Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.310181 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.310255 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310356 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.310380 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310419 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:48:55.310399312 +0000 UTC m=+3.064714192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310447 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310359 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310484 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:48:55.310471894 +0000 UTC m=+3.064786779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310488 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310502 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:54.310661 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.310534 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:48:55.310525238 +0000 UTC m=+3.064840115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:54.590781 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.590697 2519 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:54.644671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.644589 2519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:43:53 +0000 UTC" deadline="2028-03-01 18:03:33.695742353 +0000 UTC" Sep 11 13:48:54.644671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.644669 2519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12892h14m39.051077574s" Sep 11 13:48:54.685933 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.685371 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:54.685933 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:54.685528 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:48:54.697867 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.697817 2519 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:54.722220 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.722162 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvrq9" event={"ID":"b84a9e05-9178-4368-8744-1b49ee4ba041","Type":"ContainerStarted","Data":"44e396c518b22e7f052c40042ed081d854f99d0d8fe40c00cb86ff7c82a0ec4a"} Sep 11 13:48:54.740175 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.740104 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" event={"ID":"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153","Type":"ContainerStarted","Data":"c37501855305eca0aa56f0f1c3b7a2d2ed9b486127406031255f4724c94a0e49"} Sep 11 13:48:54.758093 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.758051 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpk9r" event={"ID":"5be8ff13-5b2e-409c-bebc-84a07e44ef77","Type":"ContainerStarted","Data":"57a65bd9f45c7e44c24916a8d64c0b3f997ee6e45992837d2fcc701fb4e120ae"} Sep 11 13:48:54.800031 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.799919 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc85z" event={"ID":"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8","Type":"ContainerStarted","Data":"f8d61b788085cfbafe8658d86ced0396462a2d8731fc426a859762228d4135c3"} Sep 11 13:48:54.816206 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.816097 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" event={"ID":"7095be69-101f-4123-ba41-aeaceb4eb62a","Type":"ContainerStarted","Data":"ff52c4238b4401d69be5dd8717b9e0cd5d90c73f3d9efb3582a503998579fdcb"} Sep 11 13:48:54.819880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.819837 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"b05ea7a848e1dce856ba7bd7472530f7c2a5dbea3446a542fd31bf658a633880"} Sep 11 13:48:54.841448 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.841314 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdvmj" event={"ID":"3b4a9107-64d8-45be-bfe9-ed28093f3c83","Type":"ContainerStarted","Data":"dbd320feb12cdbbbd2e706990dc6e2352c7e75df8a87b7a09b5ef0337a744034"} Sep 11 13:48:54.856809 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.856719 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerStarted","Data":"c55b376f1db47b29fb927eba972c61ba8e386d4b6ec4f47ae8efab6fe3e1c4e8"} Sep 11 13:48:54.875025 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.874944 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lngk" event={"ID":"204076d3-c263-49ce-9747-eea120329029","Type":"ContainerStarted","Data":"5e10964d6b60dd0eaf7d863f277bcef016d4328b8f356089c41321f23d44d2de"} Sep 11 13:48:54.901067 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.900979 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerStarted","Data":"3fbc590f0432a717a769a68fdf5b09559d43ddfa019c5eb042b46e401bfde176"} Sep 11 13:48:54.933169 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:54.933130 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"b23ec63d9b3d220644ea926d481ee00f","Type":"ContainerStarted","Data":"f003fa3f6d2b6366bfaa9bd0d397469f3c7448c46c556df1a700e57a26b3f077"} Sep 11 13:48:55.317455 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.317373 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:55.317455 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.317429 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:55.317724 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.317464 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:55.317724 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.317626 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:55.317724 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.317687 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:48:57.317668804 +0000 UTC m=+5.071983678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:55.318228 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318088 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:48:55.318228 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318110 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:48:55.318228 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318125 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:55.318228 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318172 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:48:57.318157236 +0000 UTC m=+5.072472110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:55.318462 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318235 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:55.318462 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.318269 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:48:57.318258001 +0000 UTC m=+5.072572876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:55.644950 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.644855 2519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:43:53 +0000 UTC" deadline="2028-04-18 14:18:12.38922517 +0000 UTC" Sep 11 13:48:55.644950 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.644898 2519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14040h29m16.74433213s" Sep 11 13:48:55.685301 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.685273 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:55.685472 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.685399 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:48:55.685851 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:55.685830 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:55.685961 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:55.685928 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:48:56.309843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:56.309812 2519 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:48:56.689443 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:56.686688 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:56.689443 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:56.686830 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:48:57.333722 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:57.333423 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:57.333722 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:57.333715 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:57.333942 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:57.333759 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:57.333942 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.333609 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:48:57.333942 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.333895 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:48:57.333942 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.333909 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:57.334126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.333966 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:49:01.333947768 +0000 UTC m=+9.088262640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:48:57.334126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.334039 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:57.334126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.334070 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:49:01.334059602 +0000 UTC m=+9.088374476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:48:57.334126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.333870 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:57.334126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.334102 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:49:01.334094022 +0000 UTC m=+9.088408906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:48:57.685475 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:57.685398 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:57.685918 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.685527 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:48:57.685918 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:57.685399 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:57.685918 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:57.685729 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:48:58.690143 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:58.690109 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:48:58.690622 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:58.690252 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:48:59.684836 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:59.684800 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:48:59.685010 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:48:59.684856 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:48:59.685010 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:59.684940 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:48:59.685099 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:48:59.685051 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:00.684672 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:00.684641 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:00.685126 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:00.684767 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:01.372040 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:01.371990 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:01.372040 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:01.372040 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:01.372256 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:01.372099 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:01.372256 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372241 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:01.372369 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372298 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:49:09.372279287 +0000 UTC m=+17.126594176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:01.372608 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372512 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:01.372608 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372566 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:49:09.372548431 +0000 UTC m=+17.126863306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:01.372608 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372569 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:49:01.372608 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372609 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:49:01.372909 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372622 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:01.372909 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.372677 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:49:09.37266183 +0000 UTC m=+17.126976712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:01.685666 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:01.685241 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:01.685666 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:01.685241 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:01.685666 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.685376 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:01.685666 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:01.685442 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:02.685952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:02.685477 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:02.685952 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:02.685613 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:03.685538 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:03.685452 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:03.685711 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:03.685462 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:03.685711 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:03.685600 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:03.685711 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:03.685664 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:04.684610 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:04.684568 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:04.685099 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:04.684718 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:05.684538 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:05.684505 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:05.684703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:05.684506 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:05.684703 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:05.684655 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:05.685049 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:05.684728 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:06.684892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:06.684858 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:06.685300 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:06.684999 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:07.685292 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:07.685254 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:07.685292 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:07.685286 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:07.685808 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:07.685381 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:07.685808 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:07.685504 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:08.684712 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:08.684677 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:08.684849 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:08.684825 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:09.432556 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:09.432517 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:09.432556 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:09.432562 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:09.432671 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432690 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432713 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432726 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432768 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432792 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:49:25.432772942 +0000 UTC m=+33.187087815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432818 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:49:25.432804893 +0000 UTC m=+33.187119768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432772 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:09.433123 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.432851 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:49:25.432841695 +0000 UTC m=+33.187156577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:09.685463 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:09.685379 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:09.685662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:09.685380 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:09.685662 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.685502 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:09.685662 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:09.685630 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:10.684935 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:10.684893 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:10.685425 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:10.685024 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:11.685393 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:11.685349 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:11.685393 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:11.685390 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:11.685888 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:11.685494 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:11.685888 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:11.685637 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:12.685300 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:12.685266 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:12.685463 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:12.685357 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:13.685313 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.685155 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:13.685429 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.685317 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:13.685429 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:13.685411 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:13.686093 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:13.685535 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:13.973278 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.973086 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" event={"ID":"9e8b0ab9-2837-4ab7-875b-9ad03ddd6153","Type":"ContainerStarted","Data":"6630f7f186566563eac7c7d81bcf0a73049fdcf72ca6d1b97a4ca4f184149943"} Sep 11 13:49:13.974700 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.974676 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc85z" event={"ID":"02e73bd8-0cd9-421c-84af-60ca5a3f3ae8","Type":"ContainerStarted","Data":"f82b68eb7795c08a708ab81238f4234fbea82f08c13d7a6fdefc029696e452fd"} Sep 11 13:49:13.976114 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.976081 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" event={"ID":"7095be69-101f-4123-ba41-aeaceb4eb62a","Type":"ContainerStarted","Data":"f7b3035172532e13ac1820f0a03286130a7ff9c74e514ee5e7ee8a76a799b357"} Sep 11 13:49:13.978763 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.978744 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:49:13.979088 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979063 2519 generic.go:356] "Generic (PLEG): container finished" podID="36998c19-023d-4d86-8313-676fd0d6527b" containerID="f274397b207f13920b8989fef82a090498bedc2f93e12e6b4edd890c6804fb3e" exitCode=1 Sep 11 13:49:13.979173 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979129 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"308f49a4c5a8500bde1ba45d22dc72c98a88076635be5c2bca5ae2178ddda87c"} Sep 11 13:49:13.979229 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979200 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"3da8da4cab1b7e4d5ec58d71a0b61e93c68114724e0995e1a51524e6f07e65db"} Sep 11 13:49:13.979274 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979253 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"e50b359316736567181e39eae7631b2de2267c88656c2910066e841e84ab3470"} Sep 11 13:49:13.979312 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979300 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"b473d393e887fe056679aa0c08620e92943126069ed47d86773f258083c1b3df"} Sep 11 13:49:13.979356 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979342 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerDied","Data":"f274397b207f13920b8989fef82a090498bedc2f93e12e6b4edd890c6804fb3e"} Sep 11 13:49:13.979400 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.979378 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"e359547f74780a6b3b71f1d8865618ace1831bca0aa6c641107d10fad4c4cf56"} Sep 11 13:49:13.980500 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.980480 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdvmj" event={"ID":"3b4a9107-64d8-45be-bfe9-ed28093f3c83","Type":"ContainerStarted","Data":"389ffbeb0ce1968d2c8beb1142dad0be033ea28a6b693d107f08904a900c38f1"} Sep 11 13:49:13.981880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.981859 2519 generic.go:356] "Generic (PLEG): container finished" podID="cef12fc99af254fa0de9a3b0e067caf7" containerID="67396419ce90e42c7482fd38f002682f03b4e73a60427d738f2cefb319b90877" exitCode=0 Sep 11 13:49:13.981946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.981921 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerDied","Data":"67396419ce90e42c7482fd38f002682f03b4e73a60427d738f2cefb319b90877"} Sep 11 13:49:13.983468 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.983451 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lngk" event={"ID":"204076d3-c263-49ce-9747-eea120329029","Type":"ContainerStarted","Data":"079c45bb464d551f312580ad028db0d862432eda6e2c5b5896ab5a735d1937e9"} Sep 11 13:49:13.984921 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.984903 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="06438b493de1b7244e124d50e68c4a4e1239d87f03acc69d77d9525dd032b987" exitCode=0 Sep 11 13:49:13.985015 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.984961 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"06438b493de1b7244e124d50e68c4a4e1239d87f03acc69d77d9525dd032b987"} Sep 11 13:49:13.986523 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.986496 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" event={"ID":"b23ec63d9b3d220644ea926d481ee00f","Type":"ContainerStarted","Data":"02922caf6c621dc840bb9ef8db64ca58b31a76edf6161d709bd14445516a4ad3"} Sep 11 13:49:13.988032 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.988012 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvrq9" event={"ID":"b84a9e05-9178-4368-8744-1b49ee4ba041","Type":"ContainerStarted","Data":"ce8c3f732b0ee0f17f1cfc7ae163a5b97b15baeda895781626477a414a5e2991"} Sep 11 13:49:13.989898 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:13.989861 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mwdc5" podStartSLOduration=2.062297892 podStartE2EDuration="20.98985105s" totalImagesPullingTime="18.927553158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:13.989664062 +0000 UTC m=+21.743978952" watchObservedRunningTime="2026-09-11 13:49:13.98985105 +0000 UTC m=+21.744165941" Sep 11 13:49:14.016944 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.016906 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-201.ec2.internal" podStartSLOduration=21.01689001 podStartE2EDuration="21.01689001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:14.016468108 +0000 UTC m=+21.770782999" watchObservedRunningTime="2026-09-11 13:49:14.01689001 +0000 UTC m=+21.771204901" Sep 11 13:49:14.030273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.030231 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lvrq9" podStartSLOduration=2.188014188 podStartE2EDuration="21.03021642s" totalImagesPullingTime="18.842202232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:14.02973532 +0000 UTC m=+21.784050211" watchObservedRunningTime="2026-09-11 13:49:14.03021642 +0000 UTC m=+21.784531310" Sep 11 13:49:14.042758 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.042713 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8lngk" podStartSLOduration=2.191048153 podStartE2EDuration="21.042699393s" totalImagesPullingTime="18.85165124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:14.042436322 +0000 UTC m=+21.796751225" watchObservedRunningTime="2026-09-11 13:49:14.042699393 +0000 UTC m=+21.797014283" Sep 11 13:49:14.057598 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.057537 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pdvmj" podStartSLOduration=3.039446569 podStartE2EDuration="22.057523195s" totalImagesPullingTime="19.018076626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:14.056912607 +0000 UTC m=+21.811227498" watchObservedRunningTime="2026-09-11 13:49:14.057523195 +0000 UTC m=+21.811838086" Sep 11 13:49:14.075109 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.075066 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vc85z" podStartSLOduration=2.5217167419999997 podStartE2EDuration="22.075053779s" totalImagesPullingTime="19.553337037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:14.074441676 +0000 UTC m=+21.828756568" watchObservedRunningTime="2026-09-11 13:49:14.075053779 +0000 UTC m=+21.829368669" Sep 11 13:49:14.084887 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.084858 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:14.162349 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.162307 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:14.264030 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.263941 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:14.325242 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.325187 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:14.684659 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.684641 2519 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 13:49:14.687655 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.687637 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:14.688227 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:14.687727 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:14.992032 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.991935 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" event={"ID":"7095be69-101f-4123-ba41-aeaceb4eb62a","Type":"ContainerStarted","Data":"e7e042eafb58856df2631e27f115300dccb31f8853c1a4f3d65bd2d6e276fe09"} Sep 11 13:49:14.993693 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.993665 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" event={"ID":"cef12fc99af254fa0de9a3b0e067caf7","Type":"ContainerStarted","Data":"aa96f3761068f0ec294447d2a024fed08231ad6ca8f5f48aade42e5df55ae132"} Sep 11 13:49:14.995119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:14.995095 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpk9r" event={"ID":"5be8ff13-5b2e-409c-bebc-84a07e44ef77","Type":"ContainerStarted","Data":"bd4d99ec00b74f9ff776141dc6b27a508086ae3b93623cc2ac748d1e9608e749"} Sep 11 13:49:15.008203 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.008156 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-201.ec2.internal" podStartSLOduration=22.008141266 podStartE2EDuration="22.008141266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:15.007914823 +0000 UTC m=+22.762229713" watchObservedRunningTime="2026-09-11 13:49:15.008141266 +0000 UTC m=+22.762456157" Sep 11 13:49:15.021117 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.021067 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zpk9r" podStartSLOduration=3.130894438 podStartE2EDuration="22.021054558s" totalImagesPullingTime="18.89016012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:15.020482341 +0000 UTC m=+22.774797233" watchObservedRunningTime="2026-09-11 13:49:15.021054558 +0000 UTC m=+22.775369450" Sep 11 13:49:15.140452 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.140402 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:15.195256 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.195220 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:15.356261 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.356185 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:49:15.356833 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.356812 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:49:15.673201 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.672894 2519 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:49:14.684655224Z","UUID":"7b49c412-7b69-4abf-9b2f-e1e682994f3f","Handler":null,"Name":"","Endpoint":""} Sep 11 13:49:15.674491 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.674473 2519 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 13:49:15.674610 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.674501 2519 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 13:49:15.684487 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.684468 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:15.684613 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:15.684468 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:15.684659 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:15.684556 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:15.684695 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:15.684649 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:16.000539 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.000410 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" event={"ID":"7095be69-101f-4123-ba41-aeaceb4eb62a","Type":"ContainerStarted","Data":"7fe3403cba6cef4c7ea308e3a7c688f9dcf48b5ab5358a91303144ef96668891"} Sep 11 13:49:16.004269 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.004246 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:49:16.004781 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.004753 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"6897c50c2a10069c5a5f99923c789ff62fc9b2e40571d13e2133211a6571d6a5"} Sep 11 13:49:16.005153 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.005135 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:49:16.005496 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.005481 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8lngk" Sep 11 13:49:16.024160 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.024108 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6smmb" podStartSLOduration=3.537248857 podStartE2EDuration="24.024093688s" totalImagesPullingTime="20.486844831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:16.023598104 +0000 UTC m=+23.777912996" watchObservedRunningTime="2026-09-11 13:49:16.024093688 +0000 UTC m=+23.778408578" Sep 11 13:49:16.086952 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.086921 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:16.685044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.685012 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:16.685316 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:16.685159 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:16.725698 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.725668 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:16.903380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.903348 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:16.983306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:16.983234 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.216664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.216629 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.273280 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.273189 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.381544 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.381497 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.478767 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.478730 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.538780 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.538704 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.652442 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.652404 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.684401 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.684370 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:17.684401 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.684400 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:17.684625 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:17.684483 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:17.684625 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:17.684610 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:17.753777 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.753745 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:17.922361 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:17.922323 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:18.025842 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:18.025786 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:18.618284 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:18.618249 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:18.659303 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:18.659271 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pdvmj_3b4a9107-64d8-45be-bfe9-ed28093f3c83/dns-node-resolver/0.log" Sep 11 13:49:18.676636 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:18.676608 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:18.685049 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:18.684668 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:18.685049 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:18.684785 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:19.017822 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.017428 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:49:19.018560 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.017879 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"ce01397ec297be17fde3b13aeebe240ade138a727836194039206a3ad9e6ba69"} Sep 11 13:49:19.459508 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.459314 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvrq9_b84a9e05-9178-4368-8744-1b49ee4ba041/node-ca/0.log" Sep 11 13:49:19.570345 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.570317 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:19.623999 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.623970 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:19.684424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.684390 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:19.684424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.684405 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:19.684617 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:19.684484 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:19.684617 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:19.684601 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:19.696540 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:19.696515 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:20.022387 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022357 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="3f5f29340db6ad92fcc35ffed0f4dadb7ed00cf86b957e4607e220c1859b791e" exitCode=0 Sep 11 13:49:20.022537 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022443 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"3f5f29340db6ad92fcc35ffed0f4dadb7ed00cf86b957e4607e220c1859b791e"} Sep 11 13:49:20.022632 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022609 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:20.022711 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022683 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:20.022765 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022726 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:20.022871 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.022855 2519 scope.go:117] "RemoveContainer" containerID="f274397b207f13920b8989fef82a090498bedc2f93e12e6b4edd890c6804fb3e" Sep 11 13:49:20.041309 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.041284 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:20.041495 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.041479 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:20.526372 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.526346 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:20.685179 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.684490 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:20.685179 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:20.684637 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:20.939557 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.939374 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkqt7"] Sep 11 13:49:20.939733 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.939680 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:20.939782 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:20.939764 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:20.942563 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.942532 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8bbp"] Sep 11 13:49:20.954344 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.954319 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6n6n"] Sep 11 13:49:20.954448 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:20.954425 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:20.954548 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:20.954527 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:21.028801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.028731 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:49:21.029189 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.029163 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" event={"ID":"36998c19-023d-4d86-8313-676fd0d6527b","Type":"ContainerStarted","Data":"3d26d5ed424f0453f71a7971e6ed696549f79c9453d8053b9b5efbe714e2ff4b"} Sep 11 13:49:21.031399 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.031370 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="772fc7e9c9401546d913d570669ee4ecf848642b04170ca83a154553488c1054" exitCode=0 Sep 11 13:49:21.031505 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.031450 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"772fc7e9c9401546d913d570669ee4ecf848642b04170ca83a154553488c1054"} Sep 11 13:49:21.031552 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.031527 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:21.031683 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:21.031664 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:21.053681 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:21.053633 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" podStartSLOduration=10.047252453 podStartE2EDuration="29.053617298s" totalImagesPullingTime="19.006364845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:21.053106148 +0000 UTC m=+28.807421039" watchObservedRunningTime="2026-09-11 13:49:21.053617298 +0000 UTC m=+28.807932180" Sep 11 13:49:22.036807 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:22.036772 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="5fdd47e392880a0ad56ef5f881bd1100e95d58f601e73868052e24b29ccda532" exitCode=0 Sep 11 13:49:22.037388 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:22.036859 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"5fdd47e392880a0ad56ef5f881bd1100e95d58f601e73868052e24b29ccda532"} Sep 11 13:49:22.685505 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:22.685470 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:22.685672 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:22.685565 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:22.685672 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:22.685643 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:22.685812 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:22.685792 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:22.685892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:22.685828 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:22.685943 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:22.685921 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:23.415829 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:23.415749 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.274928 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.274873 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.431824 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.431792 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.532742 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.532631 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.641465 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.641416 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.684794 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.684717 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:24.684794 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.684758 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:24.685015 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.684727 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:24.685015 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:24.684842 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:24.685015 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:24.684951 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:24.685177 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:24.685042 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:24.751838 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.751807 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.842616 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.842521 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:24.975209 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:24.975176 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:25.444441 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:25.444396 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:25.444441 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:25.444442 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:25.444503 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444568 2519 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444639 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444655 2519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444668 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret podName:21edc280-a387-48e2-a543-d24f2c5152cd nodeName:}" failed. No retries permitted until 2026-09-11 13:49:57.444645543 +0000 UTC m=+65.198960432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret") pod "global-pull-secret-syncer-rkqt7" (UID: "21edc280-a387-48e2-a543-d24f2c5152cd") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444660 2519 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444718 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs podName:9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2 nodeName:}" failed. No retries permitted until 2026-09-11 13:49:57.444697909 +0000 UTC m=+65.199012792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs") pod "network-metrics-daemon-p8bbp" (UID: "9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444727 2519 projected.go:196] Error preparing data for projected volume kube-api-access-l7dzx for pod openshift-network-diagnostics/network-check-target-h6n6n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:25.445001 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:25.444775 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx podName:0f380162-55d0-4763-85c2-4faf79990c2e nodeName:}" failed. No retries permitted until 2026-09-11 13:49:57.444763655 +0000 UTC m=+65.199078529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dzx" (UniqueName: "kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx") pod "network-check-target-h6n6n" (UID: "0f380162-55d0-4763-85c2-4faf79990c2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:49:25.979025 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:25.978985 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:26.169735 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.169698 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:26.372201 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.372128 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:26.685520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.685438 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:26.685520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.685466 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:26.686030 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.685439 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:26.686030 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:26.685532 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:26.686030 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:26.685622 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:26.686030 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:26.685704 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:26.770332 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.770281 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:26.845710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:26.845675 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:27.052636 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:27.052609 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerStarted","Data":"9bf45dd249f524226b546b0c8c82df99153442e4d3caf3c5c51d39da61011b46"} Sep 11 13:49:27.566857 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:27.566821 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:27.869746 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:27.869673 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:27.969829 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:27.969795 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.025323 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.025266 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.057916 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.057884 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="9bf45dd249f524226b546b0c8c82df99153442e4d3caf3c5c51d39da61011b46" exitCode=0 Sep 11 13:49:28.058054 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.057925 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"9bf45dd249f524226b546b0c8c82df99153442e4d3caf3c5c51d39da61011b46"} Sep 11 13:49:28.167762 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.167734 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.372681 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.372651 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.568753 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.568679 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.622837 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.622809 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:28.685292 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.685258 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:28.685462 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.685262 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:28.685462 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:28.685389 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:28.685462 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:28.685259 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:28.685462 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:28.685454 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:28.685695 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:28.685508 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:29.063655 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:29.063568 2519 generic.go:356] "Generic (PLEG): container finished" podID="5294fa29-44c4-427a-b6c4-5d7ef844a3a3" containerID="401b2feafd5ff5cffef1878801878e7aec1b087c4d1da333bebab9e6a80e695e" exitCode=0 Sep 11 13:49:29.063655 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:29.063607 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerDied","Data":"401b2feafd5ff5cffef1878801878e7aec1b087c4d1da333bebab9e6a80e695e"} Sep 11 13:49:29.569409 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:29.569232 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:29.764079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:29.764048 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:29.968796 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:29.968717 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:30.069638 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.069602 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8th59" event={"ID":"5294fa29-44c4-427a-b6c4-5d7ef844a3a3","Type":"ContainerStarted","Data":"18754be4b3bb8c40b12d0db323b77c5cfc85c35d08583cebc8fdd8d9429ae487"} Sep 11 13:49:30.089155 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.089112 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8th59" podStartSLOduration=7.148899242 podStartE2EDuration="37.089096778s" totalImagesPullingTime="29.940197536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:30.088864019 +0000 UTC m=+37.843178911" watchObservedRunningTime="2026-09-11 13:49:30.089096778 +0000 UTC m=+37.843411669" Sep 11 13:49:30.343350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.343273 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:30.395648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.395616 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:30.546021 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.545982 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:30.684493 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.684406 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:30.684493 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.684406 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:30.684493 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:30.684467 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:30.684784 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:30.684540 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:30.684784 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:30.684626 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:30.684784 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:30.684717 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:31.037352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.037274 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.137995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.137961 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.238181 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.238146 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.371285 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.371204 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.430163 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.430136 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.580908 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.580874 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.735198 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.735125 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.829217 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.829187 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:31.970099 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:31.970067 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:32.171477 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.171397 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:32.272448 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.272418 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:32.571602 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.571519 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:32.686190 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.686165 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:32.686363 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:32.686266 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:32.686363 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.686281 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:32.686363 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.686306 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:32.686506 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:32.686409 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:32.686557 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:32.686520 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:32.969349 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:32.969320 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.069500 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.069470 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.170253 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.170225 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.224898 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.224813 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.577749 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.577679 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.800069 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.800029 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:33.900649 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:33.900619 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:34.118241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.118207 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:34.219751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.219670 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:34.576602 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.576500 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:34.684850 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.684814 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:34.685029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.684814 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:34.685029 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:34.684921 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:34.685029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.684938 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:34.685202 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:34.685027 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:34.685202 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:34.685116 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:34.764891 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:34.764862 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:35.165120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:35.165086 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:35.369131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:35.369099 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:36.685154 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:36.685120 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:36.685154 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:36.685141 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:36.685755 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:36.685129 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:36.685755 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:36.685221 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:36.685755 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:36.685290 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:36.685755 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:36.685354 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:38.684738 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:38.684705 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:38.685099 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:38.684705 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:38.685099 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:38.684814 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:38.685099 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:38.684923 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:38.685099 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:38.684705 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:38.685099 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:38.685041 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:40.685250 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:40.685215 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:40.685662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:40.685218 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:40.685662 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:40.685343 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bbp" podUID="9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2" Sep 11 13:49:40.685662 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:40.685415 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6n6n" podUID="0f380162-55d0-4763-85c2-4faf79990c2e" Sep 11 13:49:40.685662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:40.685467 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:40.685662 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:40.685519 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rkqt7" podUID="21edc280-a387-48e2-a543-d24f2c5152cd" Sep 11 13:49:40.956600 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:40.956506 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.050281 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.050253 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.097811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.097785 2519 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-201.ec2.internal" event="NodeReady" Sep 11 13:49:41.097927 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.097890 2519 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 13:49:41.183963 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.183937 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.186540 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.186519 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jkd"] Sep 11 13:49:41.190840 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.190818 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.191029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.190998 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pzvrs"] Sep 11 13:49:41.193084 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.193052 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-klbks\"" Sep 11 13:49:41.193084 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.193077 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 13:49:41.193231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.193151 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 13:49:41.193231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.193188 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 13:49:41.193407 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.193394 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 13:49:41.198750 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.198732 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.204930 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.204886 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 13:49:41.205043 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.204952 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 13:49:41.205127 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.205108 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xnqjg\"" Sep 11 13:49:41.205740 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.205717 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jkd"] Sep 11 13:49:41.207049 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.207009 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 13:49:41.209776 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.209754 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzvrs"] Sep 11 13:49:41.280271 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.280241 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w25ml"] Sep 11 13:49:41.292948 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.292925 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w25ml"] Sep 11 13:49:41.293111 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.293008 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.295362 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.295337 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 13:49:41.295362 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.295353 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q92jh\"" Sep 11 13:49:41.295710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.295690 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 13:49:41.297077 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.297053 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.366218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366184 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx5rh\" (UniqueName: \"kubernetes.io/projected/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-api-access-tx5rh\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.366424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366237 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.366424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366260 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b99e603-b615-4339-bded-ee7639b7f5ce-cert\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.366424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366339 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.366424 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366382 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-crio-socket\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.366630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366434 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-data-volume\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.366630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.366458 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfds\" (UniqueName: \"kubernetes.io/projected/5b99e603-b615-4339-bded-ee7639b7f5ce-kube-api-access-zzfds\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.388279 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.388248 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.461951 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.461878 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:41.466947 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.466929 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e83873e-486e-49b2-9db7-f879d795ae28-metrics-tls\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.467025 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.466953 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e83873e-486e-49b2-9db7-f879d795ae28-tmp-dir\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.467025 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.466977 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467088 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467076 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-crio-socket\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467134 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467123 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-data-volume\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467170 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467142 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfds\" (UniqueName: \"kubernetes.io/projected/5b99e603-b615-4339-bded-ee7639b7f5ce-kube-api-access-zzfds\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.467206 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467170 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx5rh\" (UniqueName: \"kubernetes.io/projected/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-api-access-tx5rh\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467197 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-crio-socket\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467214 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467233 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b99e603-b615-4339-bded-ee7639b7f5ce-cert\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.467376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467263 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83873e-486e-49b2-9db7-f879d795ae28-config-volume\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.467376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467300 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmgp\" (UniqueName: \"kubernetes.io/projected/9e83873e-486e-49b2-9db7-f879d795ae28-kube-api-access-lvmgp\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.467526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467510 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-data-volume\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.467619 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.467562 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.470864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.470843 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.470944 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.470912 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b99e603-b615-4339-bded-ee7639b7f5ce-cert\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.477737 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.477713 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfds\" (UniqueName: \"kubernetes.io/projected/5b99e603-b615-4339-bded-ee7639b7f5ce-kube-api-access-zzfds\") pod \"ingress-canary-pzvrs\" (UID: \"5b99e603-b615-4339-bded-ee7639b7f5ce\") " pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.477956 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.477939 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx5rh\" (UniqueName: \"kubernetes.io/projected/3e744bc4-f55d-466a-9d34-95d2cc16bf2f-kube-api-access-tx5rh\") pod \"insights-runtime-extractor-r8jkd\" (UID: \"3e744bc4-f55d-466a-9d34-95d2cc16bf2f\") " pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.500013 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.499992 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r8jkd" Sep 11 13:49:41.506110 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.506093 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzvrs" Sep 11 13:49:41.568687 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.568603 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83873e-486e-49b2-9db7-f879d795ae28-config-volume\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.568687 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.568650 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmgp\" (UniqueName: \"kubernetes.io/projected/9e83873e-486e-49b2-9db7-f879d795ae28-kube-api-access-lvmgp\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.568687 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.568684 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e83873e-486e-49b2-9db7-f879d795ae28-metrics-tls\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.568918 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.568706 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e83873e-486e-49b2-9db7-f879d795ae28-tmp-dir\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.569050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.569029 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e83873e-486e-49b2-9db7-f879d795ae28-tmp-dir\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.569238 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.569207 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e83873e-486e-49b2-9db7-f879d795ae28-config-volume\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.571119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.571096 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e83873e-486e-49b2-9db7-f879d795ae28-metrics-tls\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.577641 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.577621 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmgp\" (UniqueName: \"kubernetes.io/projected/9e83873e-486e-49b2-9db7-f879d795ae28-kube-api-access-lvmgp\") pod \"dns-default-w25ml\" (UID: \"9e83873e-486e-49b2-9db7-f879d795ae28\") " pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.601938 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.601910 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:41.808378 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.808311 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r8jkd"] Sep 11 13:49:41.811417 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.811392 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzvrs"] Sep 11 13:49:41.812172 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:41.812148 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e744bc4_f55d_466a_9d34_95d2cc16bf2f.slice/crio-5502e12ed550cd0ba488ee78fd09a5654a50b810f89f4af945bb4cc3eb581346 WatchSource:0}: Error finding container 5502e12ed550cd0ba488ee78fd09a5654a50b810f89f4af945bb4cc3eb581346: Status 404 returned error can't find the container with id 5502e12ed550cd0ba488ee78fd09a5654a50b810f89f4af945bb4cc3eb581346 Sep 11 13:49:41.814871 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:41.814849 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b99e603_b615_4339_bded_ee7639b7f5ce.slice/crio-cd59dac81d5bfbbf37d9ef5514a05da76b0781076b3d73f16f4f89f89b85e792 WatchSource:0}: Error finding container cd59dac81d5bfbbf37d9ef5514a05da76b0781076b3d73f16f4f89f89b85e792: Status 404 returned error can't find the container with id cd59dac81d5bfbbf37d9ef5514a05da76b0781076b3d73f16f4f89f89b85e792 Sep 11 13:49:41.887588 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:41.887545 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w25ml"] Sep 11 13:49:41.890179 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:41.890156 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e83873e_486e_49b2_9db7_f879d795ae28.slice/crio-0a695c59bf55b297058be336483a25df74d882b455b903e272cba9c55185eaf5 WatchSource:0}: Error finding container 0a695c59bf55b297058be336483a25df74d882b455b903e272cba9c55185eaf5: Status 404 returned error can't find the container with id 0a695c59bf55b297058be336483a25df74d882b455b903e272cba9c55185eaf5 Sep 11 13:49:42.006168 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.006112 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:42.068176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.068089 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:42.097450 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.097424 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w25ml" event={"ID":"9e83873e-486e-49b2-9db7-f879d795ae28","Type":"ContainerStarted","Data":"0a695c59bf55b297058be336483a25df74d882b455b903e272cba9c55185eaf5"} Sep 11 13:49:42.098613 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.098591 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzvrs" event={"ID":"5b99e603-b615-4339-bded-ee7639b7f5ce","Type":"ContainerStarted","Data":"cd59dac81d5bfbbf37d9ef5514a05da76b0781076b3d73f16f4f89f89b85e792"} Sep 11 13:49:42.099946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.099927 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jkd" event={"ID":"3e744bc4-f55d-466a-9d34-95d2cc16bf2f","Type":"ContainerStarted","Data":"bf3fdd9243848450dbaf0d06f7de82fea4899bba515f2b2004b70e3fe6254452"} Sep 11 13:49:42.100036 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.099964 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jkd" event={"ID":"3e744bc4-f55d-466a-9d34-95d2cc16bf2f","Type":"ContainerStarted","Data":"5502e12ed550cd0ba488ee78fd09a5654a50b810f89f4af945bb4cc3eb581346"} Sep 11 13:49:42.688931 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.688906 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:42.689280 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.689117 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:42.689280 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.689131 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:42.691838 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.691794 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:49:42.692848 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.692699 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6pjch\"" Sep 11 13:49:42.692848 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.692778 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:49:42.693001 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.692699 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mlc2t\"" Sep 11 13:49:42.693001 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.692933 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 13:49:42.693105 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.693084 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 13:49:42.797919 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.797886 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:42.869836 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.869788 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:42.930152 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.930118 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:42.989356 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:42.989276 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:43.103235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:43.103200 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:43.202850 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:43.202813 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:43.514474 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:43.514442 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:43.614512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:43.614476 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:44.204026 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.203992 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:44.305109 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.305069 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:44.826111 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.826036 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:49:44.827869 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.827847 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.830048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.830025 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 13:49:44.830825 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.830783 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 13:49:44.830825 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.830815 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 13:49:44.831054 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.830855 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 13:49:44.831054 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.831036 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 13:49:44.831182 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.831112 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 13:49:44.831182 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.831153 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fd7b6\"" Sep 11 13:49:44.831273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.831252 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 13:49:44.839777 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.839752 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:49:44.994695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994489 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.994864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994711 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lg6\" (UniqueName: \"kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.994864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994764 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.994864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994851 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.995050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994872 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:44.995050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:44.994895 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.095738 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.095697 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.095920 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.095810 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.095920 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.095835 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lg6\" (UniqueName: \"kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.095920 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.095876 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.095920 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.095901 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.096129 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.096003 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.096594 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.096525 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.096706 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.096614 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.096879 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.096851 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.098639 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.098614 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.098746 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.098696 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.103625 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.103599 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lg6\" (UniqueName: \"kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6\") pod \"console-c7d967fc4-jrvkx\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.110238 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.110210 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w25ml" event={"ID":"9e83873e-486e-49b2-9db7-f879d795ae28","Type":"ContainerStarted","Data":"9fe1d450f5fc209d4d334324c7937d25dcb838f56f5d9f6f993dbe98cf3b9b56"} Sep 11 13:49:45.110354 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.110259 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w25ml" event={"ID":"9e83873e-486e-49b2-9db7-f879d795ae28","Type":"ContainerStarted","Data":"b073f68d38a4e2358c1fbc6f52ac77ff47f3334d608b71897c5d04926833c432"} Sep 11 13:49:45.110354 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.110343 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:45.112002 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.111978 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzvrs" event={"ID":"5b99e603-b615-4339-bded-ee7639b7f5ce","Type":"ContainerStarted","Data":"5c98602068a788486155ad76fdf5813631d619a2e83b1951db777df83db31e61"} Sep 11 13:49:45.113975 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.113955 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jkd" event={"ID":"3e744bc4-f55d-466a-9d34-95d2cc16bf2f","Type":"ContainerStarted","Data":"26b1753342f723ebf937f1e911f81b6168c040bb07984c0cc59a82b3e2e21e68"} Sep 11 13:49:45.127313 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.127265 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w25ml" podStartSLOduration=1.661122212 podStartE2EDuration="4.127249452s" totalImagesPullingTime="2.46612724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:45.126176795 +0000 UTC m=+52.880491688" watchObservedRunningTime="2026-09-11 13:49:45.127249452 +0000 UTC m=+52.881564344" Sep 11 13:49:45.140256 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.140217 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pzvrs" podStartSLOduration=1.597524551 podStartE2EDuration="4.140177926s" totalImagesPullingTime="2.542653375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:45.139675285 +0000 UTC m=+52.893990181" watchObservedRunningTime="2026-09-11 13:49:45.140177926 +0000 UTC m=+52.894492819" Sep 11 13:49:45.143981 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.143955 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:45.282756 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.282726 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.382629 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.382535 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.422861 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.422805 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj"] Sep 11 13:49:45.425461 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.425423 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:45.427820 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.427794 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:49:45.427944 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.427820 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 13:49:45.427944 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.427897 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-72wzr\"" Sep 11 13:49:45.433722 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.433559 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj"] Sep 11 13:49:45.453285 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.453256 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.600801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.600762 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/067a8f0b-2657-44fd-a978-152a55d372d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn4vj\" (UID: \"067a8f0b-2657-44fd-a978-152a55d372d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:45.659652 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:45.659571 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b679873_2c14_495a_96c9_7e2a668274de.slice/crio-7e9d65d77e01dd632aaa474b128b62a80125d193021b264eed4edb2c9e5ff9eb WatchSource:0}: Error finding container 7e9d65d77e01dd632aaa474b128b62a80125d193021b264eed4edb2c9e5ff9eb: Status 404 returned error can't find the container with id 7e9d65d77e01dd632aaa474b128b62a80125d193021b264eed4edb2c9e5ff9eb Sep 11 13:49:45.701953 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.701930 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/067a8f0b-2657-44fd-a978-152a55d372d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn4vj\" (UID: \"067a8f0b-2657-44fd-a978-152a55d372d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:45.704788 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.704762 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/067a8f0b-2657-44fd-a978-152a55d372d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mn4vj\" (UID: \"067a8f0b-2657-44fd-a978-152a55d372d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:45.719684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.719653 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.736327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.736306 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:45.830003 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.829975 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.931030 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.930949 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:45.999761 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:45.999732 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj"] Sep 11 13:49:46.002644 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:46.002618 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067a8f0b_2657_44fd_a978_152a55d372d8.slice/crio-36a76c47959870f9aefdea8d93e8e46b981ee5682ae0740cc599de7dee09cb15 WatchSource:0}: Error finding container 36a76c47959870f9aefdea8d93e8e46b981ee5682ae0740cc599de7dee09cb15: Status 404 returned error can't find the container with id 36a76c47959870f9aefdea8d93e8e46b981ee5682ae0740cc599de7dee09cb15 Sep 11 13:49:46.119744 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:46.119703 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" event={"ID":"067a8f0b-2657-44fd-a978-152a55d372d8","Type":"ContainerStarted","Data":"36a76c47959870f9aefdea8d93e8e46b981ee5682ae0740cc599de7dee09cb15"} Sep 11 13:49:46.122301 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:46.122261 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r8jkd" event={"ID":"3e744bc4-f55d-466a-9d34-95d2cc16bf2f","Type":"ContainerStarted","Data":"638e25c9b77be75148e06aa750c3a113ba1d9a4838d5182336ba4026e54ff8a0"} Sep 11 13:49:46.123603 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:46.123560 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d967fc4-jrvkx" event={"ID":"8b679873-2c14-495a-96c9-7e2a668274de","Type":"ContainerStarted","Data":"7e9d65d77e01dd632aaa474b128b62a80125d193021b264eed4edb2c9e5ff9eb"} Sep 11 13:49:46.141919 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:46.141864 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r8jkd" podStartSLOduration=2.69663728 podStartE2EDuration="5.141843918s" totalImagesPullingTime="2.445206638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:49:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:46.141475891 +0000 UTC m=+53.895790781" watchObservedRunningTime="2026-09-11 13:49:46.141843918 +0000 UTC m=+53.896158814" Sep 11 13:49:47.128668 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.128539 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" event={"ID":"067a8f0b-2657-44fd-a978-152a55d372d8","Type":"ContainerStarted","Data":"e70e9b4dcaa7f7a40c2cacb1a9d95f6d20ed965adcae94cc69f4c812e346ff1e"} Sep 11 13:49:47.129033 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.128990 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:47.134856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.134834 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" Sep 11 13:49:47.143870 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.143829 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mn4vj" podStartSLOduration=1.110888447 podStartE2EDuration="2.143812205s" totalImagesPullingTime="1.032923758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:47.143041531 +0000 UTC m=+54.897356423" watchObservedRunningTime="2026-09-11 13:49:47.143812205 +0000 UTC m=+54.898127097" Sep 11 13:49:47.447094 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.447059 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:47.490017 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.489983 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs"] Sep 11 13:49:47.492566 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.492542 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.494688 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.494663 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 13:49:47.494864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.494835 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-blvzv\"" Sep 11 13:49:47.494982 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.494909 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 13:49:47.495428 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.495382 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 13:49:47.495541 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.495491 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 13:49:47.495648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.495603 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 13:49:47.501729 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.501698 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs"] Sep 11 13:49:47.518750 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.518703 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:47.584117 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.584085 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:47.614993 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.614960 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.615152 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.615014 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.615152 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.615093 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.615152 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.615125 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldtv\" (UniqueName: \"kubernetes.io/projected/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-kube-api-access-nldtv\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.716061 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.715975 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.716061 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.716046 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldtv\" (UniqueName: \"kubernetes.io/projected/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-kube-api-access-nldtv\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.716287 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.716090 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.716287 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.716119 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.716908 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.716858 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.718967 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.718939 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.719111 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.719087 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.724392 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.724361 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldtv\" (UniqueName: \"kubernetes.io/projected/02e6a814-f1b8-40b3-b5e4-f83d3199a4e7-kube-api-access-nldtv\") pod \"prometheus-operator-68bcd7f987-xwqzs\" (UID: \"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:47.802957 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:47.802920 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" Sep 11 13:49:48.326242 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:48.326207 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:48.392995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:48.392966 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:48.668074 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:48.668043 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs"] Sep 11 13:49:48.670549 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:48.670518 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e6a814_f1b8_40b3_b5e4_f83d3199a4e7.slice/crio-669ff33340798807d57bd29b635db9044b198669dfd7fa6040c4a8247ae2dd7f WatchSource:0}: Error finding container 669ff33340798807d57bd29b635db9044b198669dfd7fa6040c4a8247ae2dd7f: Status 404 returned error can't find the container with id 669ff33340798807d57bd29b635db9044b198669dfd7fa6040c4a8247ae2dd7f Sep 11 13:49:49.138443 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:49.138400 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d967fc4-jrvkx" event={"ID":"8b679873-2c14-495a-96c9-7e2a668274de","Type":"ContainerStarted","Data":"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c"} Sep 11 13:49:49.139872 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:49.139835 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" event={"ID":"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7","Type":"ContainerStarted","Data":"669ff33340798807d57bd29b635db9044b198669dfd7fa6040c4a8247ae2dd7f"} Sep 11 13:49:49.155597 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:49.155526 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c7d967fc4-jrvkx" podStartSLOduration=2.384065089 podStartE2EDuration="5.155505923s" totalImagesPullingTime="2.771440834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:49.154788733 +0000 UTC m=+56.909103649" watchObservedRunningTime="2026-09-11 13:49:49.155505923 +0000 UTC m=+56.909820828" Sep 11 13:49:50.593910 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:50.593880 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:50.810310 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:50.810279 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:51.149727 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.149696 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" event={"ID":"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7","Type":"ContainerStarted","Data":"0488212c34d9b3769fd34df2caebc9ce75c29c8826fb6d16384fd1de66ad8266"} Sep 11 13:49:51.149911 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.149746 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" event={"ID":"02e6a814-f1b8-40b3-b5e4-f83d3199a4e7","Type":"ContainerStarted","Data":"df18cb996420851364cf06eb648979b5bad2ebf295167673783640e629b0b98d"} Sep 11 13:49:51.166033 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.165988 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xwqzs" podStartSLOduration=2.573370981 podStartE2EDuration="4.165973076s" totalImagesPullingTime="1.592602095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:51.164912782 +0000 UTC m=+58.919227675" watchObservedRunningTime="2026-09-11 13:49:51.165973076 +0000 UTC m=+58.920287966" Sep 11 13:49:51.593079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.593041 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:49:51.595300 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.595281 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.602333 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.602306 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 13:49:51.606485 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.606458 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:49:51.646990 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.646962 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.646990 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.646989 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.647131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.647008 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.647131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.647025 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.647131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.647062 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pt9z\" (UniqueName: \"kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.647131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.647101 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.647254 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.647145 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.708800 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.708768 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:51.748182 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748146 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748229 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748251 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748266 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748284 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748309 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pt9z\" (UniqueName: \"kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.748607 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.748362 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.749243 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.749214 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.749380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.749265 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.749380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.749334 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.749380 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.749361 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.750993 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.750969 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.751072 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.750977 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.768627 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.768603 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pt9z\" (UniqueName: \"kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z\") pod \"console-56489dc867-lcgsg\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:51.805755 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.805722 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:51.904260 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:51.904176 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:49:52.047459 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.047429 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqvz8" Sep 11 13:49:52.176996 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.176921 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:49:52.179565 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:52.179538 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9943dfc2_fe4f_4f18_8ba1_50fc6116847f.slice/crio-9512bff14f978a3494ab9599d126680f4390ad0d9f48eb690a3dad210c0d5703 WatchSource:0}: Error finding container 9512bff14f978a3494ab9599d126680f4390ad0d9f48eb690a3dad210c0d5703: Status 404 returned error can't find the container with id 9512bff14f978a3494ab9599d126680f4390ad0d9f48eb690a3dad210c0d5703 Sep 11 13:49:52.186918 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.186794 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.288270 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.288234 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.387225 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.387192 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.593653 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.593620 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.847823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.847744 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.905025 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.904994 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc"] Sep 11 13:49:52.907387 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.907361 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:52.909648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.909623 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 13:49:52.909765 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.909630 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sfc7v\"" Sep 11 13:49:52.909765 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.909633 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 13:49:52.920959 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.920933 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc"] Sep 11 13:49:52.921720 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.921699 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb"] Sep 11 13:49:52.923803 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.923782 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.925720 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.925699 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 13:49:52.925805 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.925753 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 13:49:52.926203 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.926189 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4pqhz\"" Sep 11 13:49:52.928417 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.928403 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 13:49:52.938720 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.938688 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q427l"] Sep 11 13:49:52.941142 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.941124 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb"] Sep 11 13:49:52.941255 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.941240 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.943235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.943215 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 13:49:52.943322 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.943243 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 13:49:52.943376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.943354 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 13:49:52.943425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.943397 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gfth2\"" Sep 11 13:49:52.950412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.950393 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:52.956551 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956530 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.956648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956570 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-metrics-client-ca\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.956648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956609 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:52.956648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956627 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.956751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956686 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:52.956751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956708 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-sys\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.956751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956726 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.956751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956749 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfbd\" (UniqueName: \"kubernetes.io/projected/3250f149-7c06-43de-9376-b156cdc195ba-kube-api-access-bgfbd\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:52.956896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956786 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-textfile\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.956896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956811 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.956896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956847 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9s4k\" (UniqueName: \"kubernetes.io/projected/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-api-access-q9s4k\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.956896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956868 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.956896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956884 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956907 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2rd\" (UniqueName: \"kubernetes.io/projected/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-kube-api-access-qt2rd\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956928 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956945 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-wtmp\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.956977 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.957011 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-root\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:52.957044 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:52.957027 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3250f149-7c06-43de-9376-b156cdc195ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.041126 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.041095 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.057776 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057745 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057791 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-metrics-client-ca\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057818 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057839 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057873 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057904 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-sys\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.057955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057932 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.058257 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.057968 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfbd\" (UniqueName: \"kubernetes.io/projected/3250f149-7c06-43de-9376-b156cdc195ba-kube-api-access-bgfbd\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.058257 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058005 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-textfile\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.058257 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058169 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-sys\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.058435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058112 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.058435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058376 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9s4k\" (UniqueName: \"kubernetes.io/projected/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-api-access-q9s4k\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.058435 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058403 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.058595 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058517 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-textfile\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.058763 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.058743 2519 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 13:49:53.058850 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.058811 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls podName:2ded9d1c-55fe-482c-8a34-bd8f57838d7a nodeName:}" failed. No retries permitted until 2026-09-11 13:49:53.55879156 +0000 UTC m=+61.313106451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls") pod "node-exporter-q427l" (UID: "2ded9d1c-55fe-482c-8a34-bd8f57838d7a") : secret "node-exporter-tls" not found Sep 11 13:49:53.058920 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.058851 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.059243 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.059179 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-metrics-client-ca\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.059460 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.059435 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.059542 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.059467 2519 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 11 13:49:53.059618 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.059545 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls podName:3250f149-7c06-43de-9376-b156cdc195ba nodeName:}" failed. No retries permitted until 2026-09-11 13:49:53.559508683 +0000 UTC m=+61.313823552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-tw2zc" (UID: "3250f149-7c06-43de-9376-b156cdc195ba") : secret "openshift-state-metrics-tls" not found Sep 11 13:49:53.059951 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.059925 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.059951 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.059960 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2rd\" (UniqueName: \"kubernetes.io/projected/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-kube-api-access-qt2rd\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.060156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.059979 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.060156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060006 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-wtmp\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.060156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060057 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.060156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060098 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-root\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.060156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060121 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3250f149-7c06-43de-9376-b156cdc195ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.060489 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060297 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.060489 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.060376 2519 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 13:49:53.060489 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:53.060417 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls podName:48912ae9-262e-4f14-9e8f-ef1a13eb67bd nodeName:}" failed. No retries permitted until 2026-09-11 13:49:53.560403499 +0000 UTC m=+61.314718370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-rsnpb" (UID: "48912ae9-262e-4f14-9e8f-ef1a13eb67bd") : secret "kube-state-metrics-tls" not found Sep 11 13:49:53.060709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060519 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.060709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060545 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-wtmp\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.060709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.060640 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-root\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.061136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.061112 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.061340 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.061322 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3250f149-7c06-43de-9376-b156cdc195ba-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.061463 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.061444 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.061987 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.061953 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.068970 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.068942 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfbd\" (UniqueName: \"kubernetes.io/projected/3250f149-7c06-43de-9376-b156cdc195ba-kube-api-access-bgfbd\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.069279 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.069251 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9s4k\" (UniqueName: \"kubernetes.io/projected/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-api-access-q9s4k\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.071687 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.071666 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2rd\" (UniqueName: \"kubernetes.io/projected/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-kube-api-access-qt2rd\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.147552 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.147465 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.161484 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.161450 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56489dc867-lcgsg" event={"ID":"9943dfc2-fe4f-4f18-8ba1-50fc6116847f","Type":"ContainerStarted","Data":"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e"} Sep 11 13:49:53.161703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.161522 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56489dc867-lcgsg" event={"ID":"9943dfc2-fe4f-4f18-8ba1-50fc6116847f","Type":"ContainerStarted","Data":"9512bff14f978a3494ab9599d126680f4390ad0d9f48eb690a3dad210c0d5703"} Sep 11 13:49:53.179827 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.179761 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56489dc867-lcgsg" podStartSLOduration=2.179748173 podStartE2EDuration="2.179748173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:53.179357214 +0000 UTC m=+60.933672105" watchObservedRunningTime="2026-09-11 13:49:53.179748173 +0000 UTC m=+60.934063064" Sep 11 13:49:53.255094 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.255054 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.360295 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.360263 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.456180 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.456103 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.564552 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.564514 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.564755 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.564557 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.564755 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.564611 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.566868 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.566840 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ded9d1c-55fe-482c-8a34-bd8f57838d7a-node-exporter-tls\") pod \"node-exporter-q427l\" (UID: \"2ded9d1c-55fe-482c-8a34-bd8f57838d7a\") " pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.566988 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.566970 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3250f149-7c06-43de-9376-b156cdc195ba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tw2zc\" (UID: \"3250f149-7c06-43de-9376-b156cdc195ba\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.567091 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.567069 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48912ae9-262e-4f14-9e8f-ef1a13eb67bd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rsnpb\" (UID: \"48912ae9-262e-4f14-9e8f-ef1a13eb67bd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.630774 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.630735 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.722332 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.722239 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.816134 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.816097 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" Sep 11 13:49:53.832327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.832300 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" Sep 11 13:49:53.849700 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.849664 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q427l" Sep 11 13:49:53.859063 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.859033 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:53.860975 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:53.860951 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded9d1c_55fe_482c_8a34_bd8f57838d7a.slice/crio-5d71f3c822f0d90533e5333dca601423e36a5c13718059a8175fe719db0d706c WatchSource:0}: Error finding container 5d71f3c822f0d90533e5333dca601423e36a5c13718059a8175fe719db0d706c: Status 404 returned error can't find the container with id 5d71f3c822f0d90533e5333dca601423e36a5c13718059a8175fe719db0d706c Sep 11 13:49:53.957917 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:53.957883 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:54.000629 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.000539 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:49:54.004530 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.004510 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.007060 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.007039 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 13:49:54.007060 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.007054 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 13:49:54.007325 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.007308 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.007685 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.007801 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8f9tw\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.008063 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.008079 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.008113 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 13:49:54.008218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.008122 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 13:49:54.008758 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.008745 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 13:49:54.023468 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.023443 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:49:54.050639 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.050607 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:54.068917 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.068888 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-web-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.068927 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.068945 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069031 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069047 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069211 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069068 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069211 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069093 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069211 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069142 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069211 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069172 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069208 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069240 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-out\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069255 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.069350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.069277 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tsp\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-kube-api-access-z9tsp\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.088737 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.088711 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc"] Sep 11 13:49:54.091274 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:54.091250 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250f149_7c06_43de_9376_b156cdc195ba.slice/crio-5cd774eab0272bf43f65f3b359bc756b4e0c4443dfd8ec0308215496a801774e WatchSource:0}: Error finding container 5cd774eab0272bf43f65f3b359bc756b4e0c4443dfd8ec0308215496a801774e: Status 404 returned error can't find the container with id 5cd774eab0272bf43f65f3b359bc756b4e0c4443dfd8ec0308215496a801774e Sep 11 13:49:54.112849 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.112825 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb"] Sep 11 13:49:54.119677 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:54.119649 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48912ae9_262e_4f14_9e8f_ef1a13eb67bd.slice/crio-120488288be5e1c4ede5e7b7349997461a3d4aa83036b554edbfb14e7e812afc WatchSource:0}: Error finding container 120488288be5e1c4ede5e7b7349997461a3d4aa83036b554edbfb14e7e812afc: Status 404 returned error can't find the container with id 120488288be5e1c4ede5e7b7349997461a3d4aa83036b554edbfb14e7e812afc Sep 11 13:49:54.166074 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.166042 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q427l" event={"ID":"2ded9d1c-55fe-482c-8a34-bd8f57838d7a","Type":"ContainerStarted","Data":"5d71f3c822f0d90533e5333dca601423e36a5c13718059a8175fe719db0d706c"} Sep 11 13:49:54.167772 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.167743 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" event={"ID":"3250f149-7c06-43de-9376-b156cdc195ba","Type":"ContainerStarted","Data":"1cc14c10e980d5ca389ee32fb53a63c8d2cb5d030f85ff426a45b2629e6aa26b"} Sep 11 13:49:54.167869 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.167810 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" event={"ID":"3250f149-7c06-43de-9376-b156cdc195ba","Type":"ContainerStarted","Data":"5cd774eab0272bf43f65f3b359bc756b4e0c4443dfd8ec0308215496a801774e"} Sep 11 13:49:54.168990 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.168963 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" event={"ID":"48912ae9-262e-4f14-9e8f-ef1a13eb67bd","Type":"ContainerStarted","Data":"120488288be5e1c4ede5e7b7349997461a3d4aa83036b554edbfb14e7e812afc"} Sep 11 13:49:54.170120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.169618 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-out\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170008 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170073 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tsp\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-kube-api-access-z9tsp\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170108 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-web-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170135 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170159 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170256 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170282 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170314 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.170617 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170359 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.170717 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:49:54.170880 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle podName:e75d638c-16e1-41a2-92c8-cabb1f34c1fc nodeName:}" failed. No retries permitted until 2026-09-11 13:49:54.670846431 +0000 UTC m=+62.425161302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e75d638c-16e1-41a2-92c8-cabb1f34c1fc") : configmap references non-existent config key: ca-bundle.crt Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.171005 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.171046 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.171090 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.171792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.171432 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.172470 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.172442 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-out\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.174060 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.174032 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.174290 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.174262 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.174397 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.174372 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.174771 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.174745 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-web-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.174921 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.174902 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.175122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.175078 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.176088 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.176068 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.176279 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.176260 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.179203 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.179176 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:54.179663 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.179647 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tsp\" (UniqueName: \"kubernetes.io/projected/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-kube-api-access-z9tsp\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.676992 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.676953 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.678107 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.678077 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75d638c-16e1-41a2-92c8-cabb1f34c1fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e75d638c-16e1-41a2-92c8-cabb1f34c1fc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:54.925770 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:54.923753 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:49:55.126785 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.126754 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w25ml" Sep 11 13:49:55.144117 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.144080 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:55.144291 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.144143 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:55.150052 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.150021 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:55.175911 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.175871 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" event={"ID":"3250f149-7c06-43de-9376-b156cdc195ba","Type":"ContainerStarted","Data":"c17e04d8002ae36f1b34b4693f67c09d4df386870324eb8178579e1b0f0048d2"} Sep 11 13:49:55.178119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.178086 2519 generic.go:356] "Generic (PLEG): container finished" podID="2ded9d1c-55fe-482c-8a34-bd8f57838d7a" containerID="a6cd8ff98d70ec6219906c6f68e57a1f172a3ad40c96129f6cca2ab3dd0834ba" exitCode=0 Sep 11 13:49:55.179127 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.178889 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q427l" event={"ID":"2ded9d1c-55fe-482c-8a34-bd8f57838d7a","Type":"ContainerDied","Data":"a6cd8ff98d70ec6219906c6f68e57a1f172a3ad40c96129f6cca2ab3dd0834ba"} Sep 11 13:49:55.184353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.184332 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:49:55.213405 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.213379 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:49:55.277533 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.277502 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:55.377743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.377669 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:55.632748 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:55.632717 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75d638c_16e1_41a2_92c8_cabb1f34c1fc.slice/crio-13e5410a61063948a8464bfcde5448bf082fb45c07ebfa5d670dbf56c5a345cf WatchSource:0}: Error finding container 13e5410a61063948a8464bfcde5448bf082fb45c07ebfa5d670dbf56c5a345cf: Status 404 returned error can't find the container with id 13e5410a61063948a8464bfcde5448bf082fb45c07ebfa5d670dbf56c5a345cf Sep 11 13:49:55.807364 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.807274 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:55.921147 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.921021 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:55.991163 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.991122 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5"] Sep 11 13:49:55.993916 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.993887 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:55.996342 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996313 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 13:49:55.996342 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996336 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3q5r2k43lihod\"" Sep 11 13:49:55.996880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996560 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 13:49:55.996880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996702 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 13:49:55.996880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996702 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-85gsz\"" Sep 11 13:49:55.996880 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996785 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 13:49:55.997114 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:55.996888 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 13:49:56.009161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.009079 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5"] Sep 11 13:49:56.044171 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.044138 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:56.091625 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.091585 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.091883 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.091857 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092005 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.091891 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-grpc-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092005 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.091931 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092005 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.091967 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xqc\" (UniqueName: \"kubernetes.io/projected/e77445f2-60d8-436b-a2e7-89a26e2727cb-kube-api-access-m5xqc\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.092049 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77445f2-60d8-436b-a2e7-89a26e2727cb-metrics-client-ca\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.092123 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.092161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.092153 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.184341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.184250 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"13e5410a61063948a8464bfcde5448bf082fb45c07ebfa5d670dbf56c5a345cf"} Sep 11 13:49:56.187505 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.187462 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q427l" event={"ID":"2ded9d1c-55fe-482c-8a34-bd8f57838d7a","Type":"ContainerStarted","Data":"2104c28fc9c1024af9545cf9d5de0a89aaa14f6fa0de5c43acbf9132bc793c6c"} Sep 11 13:49:56.187654 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.187516 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q427l" event={"ID":"2ded9d1c-55fe-482c-8a34-bd8f57838d7a","Type":"ContainerStarted","Data":"4ec7cb5a762f47779605e994973bbbe3151a1274ecf6ba1ceb7262cd8a466a58"} Sep 11 13:49:56.191800 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.191111 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" event={"ID":"3250f149-7c06-43de-9376-b156cdc195ba","Type":"ContainerStarted","Data":"7996ed50cae09c4fb58227bf06529d13a4fdc29aea039ec4a2cb665acde7a7e6"} Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192601 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192642 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192679 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-grpc-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192717 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192741 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xqc\" (UniqueName: \"kubernetes.io/projected/e77445f2-60d8-436b-a2e7-89a26e2727cb-kube-api-access-m5xqc\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.192811 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192776 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77445f2-60d8-436b-a2e7-89a26e2727cb-metrics-client-ca\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.193169 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192867 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.193169 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.192904 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.194002 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.193978 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e77445f2-60d8-436b-a2e7-89a26e2727cb-metrics-client-ca\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.196297 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.196267 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" event={"ID":"48912ae9-262e-4f14-9e8f-ef1a13eb67bd","Type":"ContainerStarted","Data":"633cd121bc700c047fb7ee14175ff40456e8c1aee61b3aff84c565b2d67dbdd6"} Sep 11 13:49:56.196399 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.196327 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" event={"ID":"48912ae9-262e-4f14-9e8f-ef1a13eb67bd","Type":"ContainerStarted","Data":"20ada6ec4681012ed0fbe1064c120a93cdddabe2b72517b31ce1fb71b3c6498a"} Sep 11 13:49:56.196399 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.196359 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" event={"ID":"48912ae9-262e-4f14-9e8f-ef1a13eb67bd","Type":"ContainerStarted","Data":"747190fc15a5503281f2e8e2c91e798426101384678f520a549c2896b65d23e5"} Sep 11 13:49:56.196632 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.196610 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.196708 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.196650 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.197550 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.197505 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.197679 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.197662 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-grpc-tls\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.197780 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.197759 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.198105 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.198086 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e77445f2-60d8-436b-a2e7-89a26e2727cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.202162 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.202141 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xqc\" (UniqueName: \"kubernetes.io/projected/e77445f2-60d8-436b-a2e7-89a26e2727cb-kube-api-access-m5xqc\") pod \"thanos-querier-79fc48d8f6-9ptd5\" (UID: \"e77445f2-60d8-436b-a2e7-89a26e2727cb\") " pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.234845 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.234789 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rsnpb" podStartSLOduration=2.693984541 podStartE2EDuration="4.234768616s" totalImagesPullingTime="1.540784075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:56.234364791 +0000 UTC m=+63.988679706" watchObservedRunningTime="2026-09-11 13:49:56.234768616 +0000 UTC m=+63.989083506" Sep 11 13:49:56.235032 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.234914 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q427l" podStartSLOduration=3.134573173 podStartE2EDuration="4.234907246s" totalImagesPullingTime="1.020363177s" totalInitContainerRuntime="79.970896ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:56.211939221 +0000 UTC m=+63.966254106" watchObservedRunningTime="2026-09-11 13:49:56.234907246 +0000 UTC m=+63.989222138" Sep 11 13:49:56.255136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.255084 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tw2zc" podStartSLOduration=2.800237828 podStartE2EDuration="4.255069834s" totalImagesPullingTime="1.454832006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:56.254532983 +0000 UTC m=+64.008847876" watchObservedRunningTime="2026-09-11 13:49:56.255069834 +0000 UTC m=+64.009384722" Sep 11 13:49:56.306042 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.306004 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:49:56.593958 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:56.593910 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5"] Sep 11 13:49:56.648201 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:56.648165 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77445f2_60d8_436b_a2e7_89a26e2727cb.slice/crio-dde7cdeed6cf4eddeb9e6e1d20e63e6a01c2bb3f82a9a0715a340ddb487b139f WatchSource:0}: Error finding container dde7cdeed6cf4eddeb9e6e1d20e63e6a01c2bb3f82a9a0715a340ddb487b139f: Status 404 returned error can't find the container with id dde7cdeed6cf4eddeb9e6e1d20e63e6a01c2bb3f82a9a0715a340ddb487b139f Sep 11 13:49:57.159998 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.159911 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9c998d659-dgjw8"] Sep 11 13:49:57.174325 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.174296 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.174844 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.174819 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9c998d659-dgjw8"] Sep 11 13:49:57.177272 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.177246 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 13:49:57.177422 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.177399 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tx4hs\"" Sep 11 13:49:57.177760 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.177737 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 13:49:57.177845 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.177816 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 13:49:57.178130 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.178115 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 13:49:57.178394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.178379 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ij2itiebk1cd\"" Sep 11 13:49:57.202171 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202141 2519 generic.go:356] "Generic (PLEG): container finished" podID="e75d638c-16e1-41a2-92c8-cabb1f34c1fc" containerID="6faade4c5aba6c64c7e491fcd311394b7a277fd7af369ecc1ee112804014965e" exitCode=0 Sep 11 13:49:57.202301 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202224 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerDied","Data":"6faade4c5aba6c64c7e491fcd311394b7a277fd7af369ecc1ee112804014965e"} Sep 11 13:49:57.202777 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202754 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-client-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.202862 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202823 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58df\" (UniqueName: \"kubernetes.io/projected/057201d7-c5b1-4380-98af-d92b2e4af41d-kube-api-access-m58df\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.202862 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202860 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-tls\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.202981 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.202914 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/057201d7-c5b1-4380-98af-d92b2e4af41d-audit-log\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.203104 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.203020 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-metrics-server-audit-profiles\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.203104 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.203064 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.203104 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.203085 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-client-certs\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.203656 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.203560 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"dde7cdeed6cf4eddeb9e6e1d20e63e6a01c2bb3f82a9a0715a340ddb487b139f"} Sep 11 13:49:57.304386 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304344 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-client-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.304598 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304505 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58df\" (UniqueName: \"kubernetes.io/projected/057201d7-c5b1-4380-98af-d92b2e4af41d-kube-api-access-m58df\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.304701 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304683 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-tls\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.304778 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304761 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/057201d7-c5b1-4380-98af-d92b2e4af41d-audit-log\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.304977 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304956 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-metrics-server-audit-profiles\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.305055 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.304999 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.305320 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.305293 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-client-certs\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.305963 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.305938 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.306091 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.306064 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/057201d7-c5b1-4380-98af-d92b2e4af41d-audit-log\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.306361 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.306331 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/057201d7-c5b1-4380-98af-d92b2e4af41d-metrics-server-audit-profiles\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.308191 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.308169 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-client-ca-bundle\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.308279 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.308205 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-client-certs\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.308355 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.308334 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/057201d7-c5b1-4380-98af-d92b2e4af41d-secret-metrics-server-tls\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.312928 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.312902 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58df\" (UniqueName: \"kubernetes.io/projected/057201d7-c5b1-4380-98af-d92b2e4af41d-kube-api-access-m58df\") pod \"metrics-server-9c998d659-dgjw8\" (UID: \"057201d7-c5b1-4380-98af-d92b2e4af41d\") " pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.406177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.406140 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:57.483975 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.483894 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:49:57.506097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.506061 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:57.506412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.506365 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:57.506540 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.506420 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:57.506540 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.506456 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:57.508856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.508837 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:49:57.508936 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.508837 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 13:49:57.508989 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.508958 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:49:57.519546 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.519443 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 13:49:57.520120 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.520091 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21edc280-a387-48e2-a543-d24f2c5152cd-original-pull-secret\") pod \"global-pull-secret-syncer-rkqt7\" (UID: \"21edc280-a387-48e2-a543-d24f2c5152cd\") " pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:57.521013 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.520978 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2-metrics-certs\") pod \"network-metrics-daemon-p8bbp\" (UID: \"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2\") " pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:57.530634 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.530605 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dzx\" (UniqueName: \"kubernetes.io/projected/0f380162-55d0-4763-85c2-4faf79990c2e-kube-api-access-l7dzx\") pod \"network-check-target-h6n6n\" (UID: \"0f380162-55d0-4763-85c2-4faf79990c2e\") " pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:57.613934 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.613896 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w"] Sep 11 13:49:57.614948 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.614922 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:57.635769 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.635737 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w"] Sep 11 13:49:57.635956 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.635838 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.638145 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.638121 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 13:49:57.638293 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.638264 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nwnkh\"" Sep 11 13:49:57.674606 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.674557 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:57.704330 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.704298 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rkqt7" Sep 11 13:49:57.708712 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.708677 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.708877 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.708793 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ltf\" (UniqueName: \"kubernetes.io/projected/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-kube-api-access-x2ltf\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.715198 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.715164 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6pjch\"" Sep 11 13:49:57.723150 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.723124 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bbp" Sep 11 13:49:57.725253 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.725222 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mlc2t\"" Sep 11 13:49:57.733948 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.733916 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:49:57.777987 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.777219 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9c998d659-dgjw8"] Sep 11 13:49:57.797529 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.797431 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:57.810855 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.810716 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ltf\" (UniqueName: \"kubernetes.io/projected/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-kube-api-access-x2ltf\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.811258 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.811200 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.814856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.814817 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.824398 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.824346 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ltf\" (UniqueName: \"kubernetes.io/projected/f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15-kube-api-access-x2ltf\") pod \"monitoring-plugin-5d85778b9d-mns6w\" (UID: \"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.947547 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.947509 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:49:57.997433 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:57.997346 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rkqt7"] Sep 11 13:49:58.000511 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:58.000467 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21edc280_a387_48e2_a543_d24f2c5152cd.slice/crio-e6b51d53afb857d4c90578e446667a946fc33542ec9f4d17d525f1c6c4e358da WatchSource:0}: Error finding container e6b51d53afb857d4c90578e446667a946fc33542ec9f4d17d525f1c6c4e358da: Status 404 returned error can't find the container with id e6b51d53afb857d4c90578e446667a946fc33542ec9f4d17d525f1c6c4e358da Sep 11 13:49:58.040906 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.038750 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8bbp"] Sep 11 13:49:58.066070 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.065991 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6n6n"] Sep 11 13:49:58.210119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.210068 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bbp" event={"ID":"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2","Type":"ContainerStarted","Data":"7bdbd6aa16bdd935eb6d3d09df01586033114dc312225c1c9eb1d906238f7dd1"} Sep 11 13:49:58.211630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.211599 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkqt7" event={"ID":"21edc280-a387-48e2-a543-d24f2c5152cd","Type":"ContainerStarted","Data":"e6b51d53afb857d4c90578e446667a946fc33542ec9f4d17d525f1c6c4e358da"} Sep 11 13:49:58.213055 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.213025 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" event={"ID":"057201d7-c5b1-4380-98af-d92b2e4af41d","Type":"ContainerStarted","Data":"c1ba8f05f0906301db9947f2ad8a6e7ae15f8afc500a8490d12a3246ed06b2a5"} Sep 11 13:49:58.243919 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:58.243837 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w"] Sep 11 13:49:58.603035 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:58.602995 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f380162_55d0_4763_85c2_4faf79990c2e.slice/crio-3dc1f448db23745f16f05669b788e39c71c00c55cd8da3c0d3d888d120fdef19 WatchSource:0}: Error finding container 3dc1f448db23745f16f05669b788e39c71c00c55cd8da3c0d3d888d120fdef19: Status 404 returned error can't find the container with id 3dc1f448db23745f16f05669b788e39c71c00c55cd8da3c0d3d888d120fdef19 Sep 11 13:49:58.604048 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:49:58.603767 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d55d8c_ef95_4fac_ab1e_f6ef667e4a15.slice/crio-181f5c06438529c70e7e0a3a81074ea456010062e1105edec4b2a563180007d4 WatchSource:0}: Error finding container 181f5c06438529c70e7e0a3a81074ea456010062e1105edec4b2a563180007d4: Status 404 returned error can't find the container with id 181f5c06438529c70e7e0a3a81074ea456010062e1105edec4b2a563180007d4 Sep 11 13:49:59.150137 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.150053 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:49:59.155314 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.155284 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 13:49:59.159119 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.159096 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.161758 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.161733 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 13:49:59.162265 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.162042 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-10nbvu9nocu53\"" Sep 11 13:49:59.162265 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.162126 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 13:49:59.162265 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.162042 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 13:49:59.162828 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.162495 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 13:49:59.162828 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.162541 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 13:49:59.163059 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163037 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 13:49:59.163239 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163218 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 13:49:59.163338 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163305 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 13:49:59.163412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163399 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 13:49:59.163984 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163860 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 13:49:59.163984 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.163893 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 13:49:59.166218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.164090 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 13:49:59.166218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.164607 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 13:49:59.166218 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.164948 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qrts8\"" Sep 11 13:49:59.169148 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.168971 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 13:49:59.176758 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.176669 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.225941 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.225994 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226021 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226053 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226080 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985zr\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-kube-api-access-985zr\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226106 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226136 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226176 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226207 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226262 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226285 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226312 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226352 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226390 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226423 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.226884 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226447 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.235079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226484 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.235079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226551 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.235079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226606 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.235079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.226629 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.237052 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.236988 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" event={"ID":"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15","Type":"ContainerStarted","Data":"181f5c06438529c70e7e0a3a81074ea456010062e1105edec4b2a563180007d4"} Sep 11 13:49:59.241136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.241068 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6n6n" event={"ID":"0f380162-55d0-4763-85c2-4faf79990c2e","Type":"ContainerStarted","Data":"3dc1f448db23745f16f05669b788e39c71c00c55cd8da3c0d3d888d120fdef19"} Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327568 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327670 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327729 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327756 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327782 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327811 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327840 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327865 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327890 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327922 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.327976 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328003 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328026 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328060 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328092 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328116 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.328645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328139 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.329596 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328168 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985zr\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-kube-api-access-985zr\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.329596 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328196 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.329596 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.328236 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.330522 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.330950 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.331110 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.331593 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.331933 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.335066 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.338696 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.338784 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.338935 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.339438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.339426 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.340071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.339632 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b94830f2-bad7-47f8-9015-aa8db8b6d54d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.340071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.339819 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.340756 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.340713 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.341056 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.341023 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.342897 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.342855 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.349037 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.348968 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.349304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.349278 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.350748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.350445 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.350748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.350589 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b94830f2-bad7-47f8-9015-aa8db8b6d54d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.358804 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.358543 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-985zr\" (UniqueName: \"kubernetes.io/projected/b94830f2-bad7-47f8-9015-aa8db8b6d54d-kube-api-access-985zr\") pod \"prometheus-k8s-0\" (UID: \"b94830f2-bad7-47f8-9015-aa8db8b6d54d\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.478434 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.478229 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:49:59.818108 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:49:59.817969 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 13:50:00.217860 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.217805 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:00.250173 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.250074 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"e9f7c50456996ce817ff406520561711c3dceee538c9ecb1de5cbffc87094590"} Sep 11 13:50:00.250173 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.250164 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"5f70db29c1151a8650ded507d13ea7f7ac13eaa5ce016a2f1bc63ea2e9ff6197"} Sep 11 13:50:00.250721 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.250205 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"a07798c171ce1b06e7f6be096bb2907b25ce84aaaec17da36a0edbcaa5f9477f"} Sep 11 13:50:00.254895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.254774 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"04fe2adb3f71e7adf3890201adf3a244438e3e9ee8b3c7ea4bf0330675bcd58e"} Sep 11 13:50:00.254895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.254851 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"277d727636f1ac0d0b753684c5c6c7bc0261831820935c657ef27f04a5de05c2"} Sep 11 13:50:00.254895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.254895 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"a70b197b2b7380c4a46e6357459d452efa06b1764cc339862c02cc452a752790"} Sep 11 13:50:00.255128 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.254933 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"f34cdb27e6f3d3952fc7a5c7ffa4278771550eecd1f615061e3bd4ff9c4e442c"} Sep 11 13:50:00.318710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.318670 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:00.630877 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.630812 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:00.645647 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.645418 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:50:00.684491 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.684461 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:50:00.687223 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.687197 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.708311 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.708277 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:50:00.745026 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.744985 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745066 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745094 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745167 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745204 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745190 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hwv\" (UniqueName: \"kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745220 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.745425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.745279 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.845935 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.845895 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.845944 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hwv\" (UniqueName: \"kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.845981 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.846039 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.846074 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846333 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.846127 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846333 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.846153 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.846877 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.846841 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:00.847196 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.847171 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.847677 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.847642 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.847677 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.847656 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.848136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.848114 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.848946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.848922 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.849556 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.849535 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:00.855591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:00.855549 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hwv\" (UniqueName: \"kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv\") pod \"console-66d68cc55c-pjjdk\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:01.000891 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:01.000804 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:01.057409 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:50:01.057369 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94830f2_bad7_47f8_9015_aa8db8b6d54d.slice/crio-e1737a87a27eeec8e02ade45358759ac4eb051a3dfc9ad54c9006f8cae54cc92 WatchSource:0}: Error finding container e1737a87a27eeec8e02ade45358759ac4eb051a3dfc9ad54c9006f8cae54cc92: Status 404 returned error can't find the container with id e1737a87a27eeec8e02ade45358759ac4eb051a3dfc9ad54c9006f8cae54cc92 Sep 11 13:50:01.263665 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:01.263558 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"e1737a87a27eeec8e02ade45358759ac4eb051a3dfc9ad54c9006f8cae54cc92"} Sep 11 13:50:01.578402 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:01.578361 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:50:01.904570 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:01.904514 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:50:02.277622 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.277515 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"5c0525db55b75f6fc7fca1c338b982c699ba51857edd9a12ab673c7ef56c4645"} Sep 11 13:50:02.277622 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.277608 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"9eb3f1efdc42c0b27df8babc8b4f3f1f77919208d3a85d0d5cf4285e419c745b"} Sep 11 13:50:02.278189 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.277639 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" event={"ID":"e77445f2-60d8-436b-a2e7-89a26e2727cb","Type":"ContainerStarted","Data":"071f85d8bd7058b8cf39344c0ff7efe01c7fb98768c12cd700ed66710ec33e1b"} Sep 11 13:50:02.278189 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.277733 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:50:02.284642 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.284607 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"7e8b7e17f47f608ea42f463f0926af8e8fb243132d0e3dadd6e7a037974da9c9"} Sep 11 13:50:02.284789 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.284686 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e75d638c-16e1-41a2-92c8-cabb1f34c1fc","Type":"ContainerStarted","Data":"12501dfd8544a9748290e2f79ea0922a62c4bfdef1cc75b0c03aad57c7b13d53"} Sep 11 13:50:02.286655 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.286608 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d68cc55c-pjjdk" event={"ID":"201c55d1-b1fb-4a6c-a4a6-504a056057a5","Type":"ContainerStarted","Data":"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a"} Sep 11 13:50:02.286772 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.286670 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d68cc55c-pjjdk" event={"ID":"201c55d1-b1fb-4a6c-a4a6-504a056057a5","Type":"ContainerStarted","Data":"2f6c2874d803ca9e08aa36ae75ce6a04e568189cca083e2cc57a61033a896f59"} Sep 11 13:50:02.288877 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.288851 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bbp" event={"ID":"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2","Type":"ContainerStarted","Data":"9c2cb50ead235416dc78f220d815d5e540c2a8d97df0121967367156d34c7e19"} Sep 11 13:50:02.289003 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.288902 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bbp" event={"ID":"9d6c88a8-3ef4-41f1-9386-7cd7795ec5c2","Type":"ContainerStarted","Data":"0d247de27070bd6e5a722e68b1372ac62f424325c69065927ec3940c66a91562"} Sep 11 13:50:02.290534 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.290511 2519 generic.go:356] "Generic (PLEG): container finished" podID="b94830f2-bad7-47f8-9015-aa8db8b6d54d" containerID="86834047790db94c78f6a168b06196c5900b138c2980d469ddc81c22c774ebec" exitCode=0 Sep 11 13:50:02.290656 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.290598 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerDied","Data":"86834047790db94c78f6a168b06196c5900b138c2980d469ddc81c22c774ebec"} Sep 11 13:50:02.292417 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.292392 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" event={"ID":"f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15","Type":"ContainerStarted","Data":"a7a8ad1f8072a3a2572e542d4c0a9882299a7a68f95488f9012357e36d601af5"} Sep 11 13:50:02.292733 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.292706 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:50:02.294252 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.294192 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" event={"ID":"057201d7-c5b1-4380-98af-d92b2e4af41d","Type":"ContainerStarted","Data":"b00767ce82fbb9bbff44b32ae5befabc162ce08d67230dcc0c34051b74a95bdc"} Sep 11 13:50:02.298942 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.298916 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" Sep 11 13:50:02.305912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.305365 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" podStartSLOduration=3.043219669 podStartE2EDuration="7.305346682s" totalImagesPullingTime="4.262127013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.303814417 +0000 UTC m=+70.058129314" watchObservedRunningTime="2026-09-11 13:50:02.305346682 +0000 UTC m=+70.059661584" Sep 11 13:50:02.333482 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.333428 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.136136731 podStartE2EDuration="9.333412128s" totalImagesPullingTime="3.179782728s" totalInitContainerRuntime="17.492669ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.332065073 +0000 UTC m=+70.086379965" watchObservedRunningTime="2026-09-11 13:50:02.333412128 +0000 UTC m=+70.087727019" Sep 11 13:50:02.350625 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.350542 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" podStartSLOduration=1.828428172 podStartE2EDuration="5.350520652s" totalImagesPullingTime="3.52209248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.349769788 +0000 UTC m=+70.104084679" watchObservedRunningTime="2026-09-11 13:50:02.350520652 +0000 UTC m=+70.104835569" Sep 11 13:50:02.384375 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.384317 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d68cc55c-pjjdk" podStartSLOduration=2.384287157 podStartE2EDuration="2.384287157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.367789951 +0000 UTC m=+70.122104842" watchObservedRunningTime="2026-09-11 13:50:02.384287157 +0000 UTC m=+70.138602047" Sep 11 13:50:02.386075 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.386020 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8bbp" podStartSLOduration=67.115345148 podStartE2EDuration="1m10.386002222s" totalImagesPullingTime="3.270657074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.384236238 +0000 UTC m=+70.138551130" watchObservedRunningTime="2026-09-11 13:50:02.386002222 +0000 UTC m=+70.140317114" Sep 11 13:50:02.401154 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.401092 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mns6w" podStartSLOduration=2.69566286 podStartE2EDuration="5.401069591s" totalImagesPullingTime="2.705406731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:02.400617044 +0000 UTC m=+70.154931936" watchObservedRunningTime="2026-09-11 13:50:02.401069591 +0000 UTC m=+70.155384481" Sep 11 13:50:02.507729 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.507692 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:02.602819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:02.602781 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:04.713110 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:04.713018 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:04.764463 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:04.764423 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:04.830426 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:04.830389 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:04.887512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:04.887477 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:05.311188 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:05.311150 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rkqt7" event={"ID":"21edc280-a387-48e2-a543-d24f2c5152cd","Type":"ContainerStarted","Data":"4040b895f86effb8300b2c8224f741f8913f712d22b9de30f0b85c3bac0607e4"} Sep 11 13:50:05.313122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:05.313091 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6n6n" event={"ID":"0f380162-55d0-4763-85c2-4faf79990c2e","Type":"ContainerStarted","Data":"b0ade7b19c7e63b5e95a547be6e3c6d3b2da2b84326346f2dcf3e1663e5e3675"} Sep 11 13:50:05.313258 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:05.313233 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:50:05.327316 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:05.327257 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rkqt7" podStartSLOduration=65.933753992 podStartE2EDuration="1m12.327235941s" totalImagesPullingTime="6.393481949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:05.325540377 +0000 UTC m=+73.079855271" watchObservedRunningTime="2026-09-11 13:50:05.327235941 +0000 UTC m=+73.081550849" Sep 11 13:50:05.341241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:05.341180 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h6n6n" podStartSLOduration=67.553637372 podStartE2EDuration="1m13.341160737s" totalImagesPullingTime="5.787523365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:05.339007594 +0000 UTC m=+73.093322485" watchObservedRunningTime="2026-09-11 13:50:05.341160737 +0000 UTC m=+73.095475632" Sep 11 13:50:07.327349 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327315 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"d5fcbfa6c544784dabb546212a166a5311d0af0c53c60d16ea64eeaa67981fa2"} Sep 11 13:50:07.327703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327410 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"5d731d631dcc63f89b0561ede4a8b6dddc7d165e4f499041f935f2bbf022f1e8"} Sep 11 13:50:07.327703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327452 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"b8a5d2010f34d1bf0de86eb5419a9efa571b02794f7eb9ba6e1b4f2d47e801a8"} Sep 11 13:50:07.327703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327495 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"efde57ad048cd9de4d3fb801e1b163cd10a689b1a64353fec474ab8301bd0525"} Sep 11 13:50:07.327703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327532 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"e5e8cdf1270d353be5ba5622d2cfe6e96c143195fc3302d13fedeb7c72e65959"} Sep 11 13:50:07.327703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.327566 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b94830f2-bad7-47f8-9015-aa8db8b6d54d","Type":"ContainerStarted","Data":"061d4f14661f375e84256a927b6c9e3a23b6a50d503adc92887b9ec965430a67"} Sep 11 13:50:07.359371 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:07.359321 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.693631689 podStartE2EDuration="8.359304418s" totalImagesPullingTime="4.635510282s" totalInitContainerRuntime="30.162447ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:07.356633057 +0000 UTC m=+75.110947948" watchObservedRunningTime="2026-09-11 13:50:07.359304418 +0000 UTC m=+75.113619308" Sep 11 13:50:08.305049 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:08.305020 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79fc48d8f6-9ptd5" Sep 11 13:50:08.556400 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:08.556308 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:08.656651 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:08.656615 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:08.708326 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:08.708296 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:09.478704 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:09.478663 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:50:09.675008 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:09.674974 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:09.733131 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:09.733063 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:11.001456 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.001421 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:11.001859 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.001482 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:11.006403 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.006368 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:11.348649 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.348624 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:50:11.396808 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.396769 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:50:11.499743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.499711 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:11.599653 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:11.599562 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:12.117533 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:12.117485 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:12.215029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:12.214993 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:16.017122 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:16.016993 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:16.086657 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:16.086615 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:17.484425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:17.484390 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:50:17.484953 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:17.484489 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:50:25.671241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:25.671172 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56489dc867-lcgsg" podUID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" containerName="console" containerID="cri-o://7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e" gracePeriod=15 Sep 11 13:50:25.922405 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:25.922348 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:50:26.087461 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087416 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087461 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087461 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087500 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087551 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087622 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087653 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087743 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087676 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pt9z\" (UniqueName: \"kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z\") pod \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\" (UID: \"9943dfc2-fe4f-4f18-8ba1-50fc6116847f\") " Sep 11 13:50:26.087976 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087944 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:26.087976 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.087963 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:26.088206 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.088184 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config" (OuterVolumeSpecName: "console-config") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:26.088291 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.088271 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca" (OuterVolumeSpecName: "service-ca") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:26.089926 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.089899 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z" (OuterVolumeSpecName: "kube-api-access-4pt9z") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "kube-api-access-4pt9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:50:26.090042 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.089985 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:50:26.090084 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.090045 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9943dfc2-fe4f-4f18-8ba1-50fc6116847f" (UID: "9943dfc2-fe4f-4f18-8ba1-50fc6116847f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189315 2519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189344 2519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189353 2519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189362 2519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189370 2519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189378 2519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.189390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.189387 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pt9z\" (UniqueName: \"kubernetes.io/projected/9943dfc2-fe4f-4f18-8ba1-50fc6116847f-kube-api-access-4pt9z\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:26.410483 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.410444 2519 generic.go:356] "Generic (PLEG): container finished" podID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" containerID="7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e" exitCode=0 Sep 11 13:50:26.410483 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.410487 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56489dc867-lcgsg" Sep 11 13:50:26.410708 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.410522 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56489dc867-lcgsg" event={"ID":"9943dfc2-fe4f-4f18-8ba1-50fc6116847f","Type":"ContainerDied","Data":"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e"} Sep 11 13:50:26.410708 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.410590 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56489dc867-lcgsg" event={"ID":"9943dfc2-fe4f-4f18-8ba1-50fc6116847f","Type":"ContainerDied","Data":"9512bff14f978a3494ab9599d126680f4390ad0d9f48eb690a3dad210c0d5703"} Sep 11 13:50:26.410708 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.410612 2519 scope.go:117] "RemoveContainer" containerID="7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e" Sep 11 13:50:26.420014 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.419998 2519 scope.go:117] "RemoveContainer" containerID="7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e" Sep 11 13:50:26.420305 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:50:26.420283 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e\": container with ID starting with 7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e not found: ID does not exist" containerID="7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e" Sep 11 13:50:26.420375 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.420312 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e"} err="failed to get container status \"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e\": rpc error: code = NotFound desc = could not find container \"7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e\": container with ID starting with 7c10fb4f3fbf0c3fb097bb8fdeb8de359a172bf0fb091e0368e5586567d6443e not found: ID does not exist" Sep 11 13:50:26.431290 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.431264 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:50:26.435115 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.435096 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56489dc867-lcgsg"] Sep 11 13:50:26.690221 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.690189 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" path="/var/lib/kubelet/pods/9943dfc2-fe4f-4f18-8ba1-50fc6116847f/volumes" Sep 11 13:50:26.867730 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.867699 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:26.970266 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:26.970192 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:31.028694 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:31.028660 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:31.126676 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:31.126645 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:36.323974 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.323945 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h6n6n" Sep 11 13:50:36.416297 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.416258 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c7d967fc4-jrvkx" podUID="8b679873-2c14-495a-96c9-7e2a668274de" containerName="console" containerID="cri-o://fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c" gracePeriod=15 Sep 11 13:50:36.656522 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.656497 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:50:36.673692 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673667 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.673768 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673711 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.673768 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673732 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.673768 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673753 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.673902 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673879 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94lg6\" (UniqueName: \"kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.673957 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.673936 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config\") pod \"8b679873-2c14-495a-96c9-7e2a668274de\" (UID: \"8b679873-2c14-495a-96c9-7e2a668274de\") " Sep 11 13:50:36.674174 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674121 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca" (OuterVolumeSpecName: "service-ca") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:36.674174 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674129 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:36.674345 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674174 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config" (OuterVolumeSpecName: "console-config") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:50:36.674345 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674263 2519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.674345 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674282 2519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.674345 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.674295 2519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b679873-2c14-495a-96c9-7e2a668274de-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.676282 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.676239 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:50:36.676282 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.676270 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6" (OuterVolumeSpecName: "kube-api-access-94lg6") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "kube-api-access-94lg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:50:36.676684 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.676649 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8b679873-2c14-495a-96c9-7e2a668274de" (UID: "8b679873-2c14-495a-96c9-7e2a668274de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:50:36.775664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.775630 2519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.775664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.775657 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94lg6\" (UniqueName: \"kubernetes.io/projected/8b679873-2c14-495a-96c9-7e2a668274de-kube-api-access-94lg6\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.775664 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.775667 2519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b679873-2c14-495a-96c9-7e2a668274de-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:50:36.980622 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:36.980569 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:37.082895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.082865 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:50:37.457287 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.457250 2519 generic.go:356] "Generic (PLEG): container finished" podID="8b679873-2c14-495a-96c9-7e2a668274de" containerID="fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c" exitCode=0 Sep 11 13:50:37.457719 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.457296 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7d967fc4-jrvkx" Sep 11 13:50:37.457719 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.457324 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d967fc4-jrvkx" event={"ID":"8b679873-2c14-495a-96c9-7e2a668274de","Type":"ContainerDied","Data":"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c"} Sep 11 13:50:37.457719 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.457367 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7d967fc4-jrvkx" event={"ID":"8b679873-2c14-495a-96c9-7e2a668274de","Type":"ContainerDied","Data":"7e9d65d77e01dd632aaa474b128b62a80125d193021b264eed4edb2c9e5ff9eb"} Sep 11 13:50:37.457719 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.457389 2519 scope.go:117] "RemoveContainer" containerID="fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c" Sep 11 13:50:37.469281 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.469246 2519 scope.go:117] "RemoveContainer" containerID="fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c" Sep 11 13:50:37.469714 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:50:37.469688 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c\": container with ID starting with fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c not found: ID does not exist" containerID="fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c" Sep 11 13:50:37.469801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.469721 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c"} err="failed to get container status \"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c\": rpc error: code = NotFound desc = could not find container \"fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c\": container with ID starting with fe1dfab0a312197685869e91ab6a630e1790314d846dd145883413c046d3fe4c not found: ID does not exist" Sep 11 13:50:37.474143 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.474120 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:50:37.477310 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.477289 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c7d967fc4-jrvkx"] Sep 11 13:50:37.489309 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.489285 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:50:37.492959 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:37.492939 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9c998d659-dgjw8" Sep 11 13:50:38.689804 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:38.689763 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b679873-2c14-495a-96c9-7e2a668274de" path="/var/lib/kubelet/pods/8b679873-2c14-495a-96c9-7e2a668274de/volumes" Sep 11 13:50:59.479278 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:59.479227 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:50:59.500193 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:59.500162 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:50:59.561718 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:50:59.561371 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 13:51:12.040995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:12.040956 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:12.141772 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:12.141734 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:17.125912 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.125876 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-656d85656d-jkb62"] Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126186 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b679873-2c14-495a-96c9-7e2a668274de" containerName="console" Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126197 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b679873-2c14-495a-96c9-7e2a668274de" containerName="console" Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126219 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" containerName="console" Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126224 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" containerName="console" Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126282 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b679873-2c14-495a-96c9-7e2a668274de" containerName="console" Sep 11 13:51:17.126394 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.126289 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="9943dfc2-fe4f-4f18-8ba1-50fc6116847f" containerName="console" Sep 11 13:51:17.129351 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.129327 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.131875 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.131848 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h46z5\"" Sep 11 13:51:17.132016 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.131970 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 13:51:17.132078 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.132064 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 13:51:17.132250 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.132235 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 13:51:17.132302 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.132262 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 13:51:17.132631 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.132502 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 13:51:17.138676 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.138651 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 13:51:17.142951 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.142835 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-serving-certs-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.142951 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.142882 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143249 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143433 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143417 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143542 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143528 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143683 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143667 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6x4\" (UniqueName: \"kubernetes.io/projected/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-kube-api-access-ws6x4\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143793 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143779 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-federate-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.143887 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.143874 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-metrics-client-ca\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.144062 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.144044 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-656d85656d-jkb62"] Sep 11 13:51:17.245320 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245288 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245320 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245323 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245714 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245342 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245714 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245357 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6x4\" (UniqueName: \"kubernetes.io/projected/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-kube-api-access-ws6x4\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245714 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245378 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-federate-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245714 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245394 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-metrics-client-ca\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245714 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245422 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-serving-certs-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.245966 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.245842 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.246284 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.246249 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-metrics-client-ca\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.246399 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.246250 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-serving-certs-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.246598 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.246542 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.248156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.248128 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-federate-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.248254 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.248156 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.248535 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.248517 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-telemeter-client-tls\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.248738 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.248719 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-secret-telemeter-client\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.253013 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.252991 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6x4\" (UniqueName: \"kubernetes.io/projected/fea163e9-a3cf-4bc5-b909-85c70b9cdb9a-kube-api-access-ws6x4\") pod \"telemeter-client-656d85656d-jkb62\" (UID: \"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a\") " pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.440612 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.440499 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" Sep 11 13:51:17.712895 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:17.712717 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-656d85656d-jkb62"] Sep 11 13:51:17.715757 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:51:17.715730 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea163e9_a3cf_4bc5_b909_85c70b9cdb9a.slice/crio-cbb25ef58c490295ab80b7279d58597512fc968d7b9d01ac593be1d4869c998b WatchSource:0}: Error finding container cbb25ef58c490295ab80b7279d58597512fc968d7b9d01ac593be1d4869c998b: Status 404 returned error can't find the container with id cbb25ef58c490295ab80b7279d58597512fc968d7b9d01ac593be1d4869c998b Sep 11 13:51:18.626850 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:18.626814 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" event={"ID":"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a","Type":"ContainerStarted","Data":"cbb25ef58c490295ab80b7279d58597512fc968d7b9d01ac593be1d4869c998b"} Sep 11 13:51:19.633165 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.633135 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" event={"ID":"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a","Type":"ContainerStarted","Data":"ce8af745375c4813621fb94c6b8eeebca1e44a9a73facf889d326a5b6590dde8"} Sep 11 13:51:19.633591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.633194 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" event={"ID":"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a","Type":"ContainerStarted","Data":"efece707daec0f082add3783e5474ee91722980d248103eb71a327ca9821e70f"} Sep 11 13:51:19.633591 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.633223 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" event={"ID":"fea163e9-a3cf-4bc5-b909-85c70b9cdb9a","Type":"ContainerStarted","Data":"a1ba9f3b32a40cfd727626a61540dca4604dad6eb5a4707a3b6873407982b1e4"} Sep 11 13:51:19.656426 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.656336 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-656d85656d-jkb62" podStartSLOduration=0.928151873 podStartE2EDuration="2.656318092s" totalImagesPullingTime="1.728166219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:51:19.654301754 +0000 UTC m=+147.408616645" watchObservedRunningTime="2026-09-11 13:51:19.656318092 +0000 UTC m=+147.410632982" Sep 11 13:51:19.730306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.730220 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:19.732745 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.732719 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.743736 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.743715 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:19.769420 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769392 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769528 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769428 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmphm\" (UniqueName: \"kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769528 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769446 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769528 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769472 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769651 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769629 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769692 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769661 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.769829 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.769808 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.785493 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.785461 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:19.870785 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870750 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmphm\" (UniqueName: \"kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870785 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870787 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870819 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870860 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870877 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870903 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.870995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.870940 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.871686 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.871657 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.871784 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.871697 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.871784 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.871727 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.871784 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.871754 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.873422 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.873404 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.873506 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.873488 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.878384 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.878362 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmphm\" (UniqueName: \"kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm\") pod \"console-6b79fbd5dc-dpkvm\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:19.885065 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:19.885040 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:20.041852 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:20.041781 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:20.311326 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:20.311132 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:20.313969 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:51:20.313936 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d3c288_1ccc_4647_8a51_f18e6b6c2e34.slice/crio-e43842896a85ee0a049faa9e3e86093ef48dac4ee2a6819ba04c63ada3600f3a WatchSource:0}: Error finding container e43842896a85ee0a049faa9e3e86093ef48dac4ee2a6819ba04c63ada3600f3a: Status 404 returned error can't find the container with id e43842896a85ee0a049faa9e3e86093ef48dac4ee2a6819ba04c63ada3600f3a Sep 11 13:51:20.644241 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:20.644190 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b79fbd5dc-dpkvm" event={"ID":"19d3c288-1ccc-4647-8a51-f18e6b6c2e34","Type":"ContainerStarted","Data":"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8"} Sep 11 13:51:20.644766 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:20.644316 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b79fbd5dc-dpkvm" event={"ID":"19d3c288-1ccc-4647-8a51-f18e6b6c2e34","Type":"ContainerStarted","Data":"e43842896a85ee0a049faa9e3e86093ef48dac4ee2a6819ba04c63ada3600f3a"} Sep 11 13:51:20.661589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:20.661537 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b79fbd5dc-dpkvm" podStartSLOduration=1.6615218619999998 podStartE2EDuration="1.661521862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:51:20.659261046 +0000 UTC m=+148.413575936" watchObservedRunningTime="2026-09-11 13:51:20.661521862 +0000 UTC m=+148.415836752" Sep 11 13:51:27.277146 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.277111 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:27.339207 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.339157 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:27.830683 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.830648 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:27.860862 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.860834 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:51:27.863248 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.863230 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.874980 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.874952 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:51:27.939179 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939139 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939189 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939218 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939235 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939258 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939275 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:27.939353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:27.939294 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d589v\" (UniqueName: \"kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.027681 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.027643 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:28.040480 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040452 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040567 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040489 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d589v\" (UniqueName: \"kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040567 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040543 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040594 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040659 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040674 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.040747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.040729 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.041376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.041350 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.041376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.041371 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.041569 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.041433 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.041569 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.041522 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.043103 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.043084 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.043227 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.043208 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.049324 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.049297 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d589v\" (UniqueName: \"kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v\") pod \"console-854bf86c8f-dtxfz\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.092861 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.092830 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:28.172370 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.172332 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:28.461737 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.461670 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:51:28.464986 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:51:28.464958 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2224931_7a69_4bc0_b6fe_0a93a0554e9d.slice/crio-4a6bf5c0bde70c1bc080fe391d449baf34b04a402bc1fb48a88a1872705b49e1 WatchSource:0}: Error finding container 4a6bf5c0bde70c1bc080fe391d449baf34b04a402bc1fb48a88a1872705b49e1: Status 404 returned error can't find the container with id 4a6bf5c0bde70c1bc080fe391d449baf34b04a402bc1fb48a88a1872705b49e1 Sep 11 13:51:28.638007 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.637976 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:28.681104 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.681073 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854bf86c8f-dtxfz" event={"ID":"b2224931-7a69-4bc0-b6fe-0a93a0554e9d","Type":"ContainerStarted","Data":"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2"} Sep 11 13:51:28.681268 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.681123 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854bf86c8f-dtxfz" event={"ID":"b2224931-7a69-4bc0-b6fe-0a93a0554e9d","Type":"ContainerStarted","Data":"4a6bf5c0bde70c1bc080fe391d449baf34b04a402bc1fb48a88a1872705b49e1"} Sep 11 13:51:28.697779 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.697736 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-854bf86c8f-dtxfz" podStartSLOduration=1.69772118 podStartE2EDuration="1.69772118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:51:28.695769812 +0000 UTC m=+156.450084703" watchObservedRunningTime="2026-09-11 13:51:28.69772118 +0000 UTC m=+156.452036070" Sep 11 13:51:28.811547 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:28.811448 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:30.042887 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:30.042854 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:32.398070 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:32.398021 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:32.456691 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:32.456662 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:38.172632 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.172565 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:38.173059 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.172665 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:38.177609 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.177566 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:38.723281 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.723248 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:51:38.772474 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.772439 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:51:38.871165 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.871127 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:38.926971 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.926944 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:38.990560 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:38.990488 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:39.148635 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:39.148582 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:39.212671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:39.212617 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:46.035127 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:46.035040 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:46.133596 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:46.133525 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:46.194551 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:46.194522 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:51:52.849677 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:52.849636 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b79fbd5dc-dpkvm" podUID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" containerName="console" containerID="cri-o://658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8" gracePeriod=15 Sep 11 13:51:53.090101 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.090075 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:53.172748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172666 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.172748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172708 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.172748 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172730 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmphm\" (UniqueName: \"kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.173029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172757 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.173029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172817 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.173029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172877 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.173029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.172927 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle\") pod \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\" (UID: \"19d3c288-1ccc-4647-8a51-f18e6b6c2e34\") " Sep 11 13:51:53.173237 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.173055 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca" (OuterVolumeSpecName: "service-ca") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:51:53.173237 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.173221 2519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.173384 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.173354 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:51:53.173483 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.173365 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:51:53.173656 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.173533 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config" (OuterVolumeSpecName: "console-config") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:51:53.175005 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.174977 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm" (OuterVolumeSpecName: "kube-api-access-lmphm") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "kube-api-access-lmphm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:51:53.175005 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.174992 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:51:53.175118 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.175025 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19d3c288-1ccc-4647-8a51-f18e6b6c2e34" (UID: "19d3c288-1ccc-4647-8a51-f18e6b6c2e34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:51:53.273683 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273647 2519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.273683 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273675 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmphm\" (UniqueName: \"kubernetes.io/projected/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-kube-api-access-lmphm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.273683 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273686 2519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.273908 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273695 2519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.273908 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273707 2519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.273908 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.273714 2519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d3c288-1ccc-4647-8a51-f18e6b6c2e34-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:51:53.778097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.778067 2519 generic.go:356] "Generic (PLEG): container finished" podID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" containerID="658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8" exitCode=0 Sep 11 13:51:53.778273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.778112 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b79fbd5dc-dpkvm" Sep 11 13:51:53.778273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.778135 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b79fbd5dc-dpkvm" event={"ID":"19d3c288-1ccc-4647-8a51-f18e6b6c2e34","Type":"ContainerDied","Data":"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8"} Sep 11 13:51:53.778273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.778172 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b79fbd5dc-dpkvm" event={"ID":"19d3c288-1ccc-4647-8a51-f18e6b6c2e34","Type":"ContainerDied","Data":"e43842896a85ee0a049faa9e3e86093ef48dac4ee2a6819ba04c63ada3600f3a"} Sep 11 13:51:53.778273 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.778191 2519 scope.go:117] "RemoveContainer" containerID="658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8" Sep 11 13:51:53.786657 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.786636 2519 scope.go:117] "RemoveContainer" containerID="658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8" Sep 11 13:51:53.786931 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:51:53.786911 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8\": container with ID starting with 658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8 not found: ID does not exist" containerID="658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8" Sep 11 13:51:53.786985 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.786940 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8"} err="failed to get container status \"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8\": rpc error: code = NotFound desc = could not find container \"658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8\": container with ID starting with 658021e8fc7dc46f5ce17e3950e7207404137b7e3d3cbf13c588d0a8e86ce8e8 not found: ID does not exist" Sep 11 13:51:53.795458 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.795436 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:53.798711 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:53.798692 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b79fbd5dc-dpkvm"] Sep 11 13:51:54.690106 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:51:54.690074 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" path="/var/lib/kubelet/pods/19d3c288-1ccc-4647-8a51-f18e6b6c2e34/volumes" Sep 11 13:52:01.078874 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:01.078839 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:01.134674 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:01.134639 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:03.791327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:03.791268 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66d68cc55c-pjjdk" podUID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" containerName="console" containerID="cri-o://587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a" gracePeriod=15 Sep 11 13:52:04.023320 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.023288 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:52:04.047017 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.046947 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047017 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047008 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047033 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6hwv\" (UniqueName: \"kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047084 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047108 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047145 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047182 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config\") pod \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\" (UID: \"201c55d1-b1fb-4a6c-a4a6-504a056057a5\") " Sep 11 13:52:04.047841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047494 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config" (OuterVolumeSpecName: "console-config") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:52:04.047841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047608 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca" (OuterVolumeSpecName: "service-ca") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:52:04.047841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047638 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:52:04.047841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.047814 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:52:04.049443 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.049410 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv" (OuterVolumeSpecName: "kube-api-access-r6hwv") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "kube-api-access-r6hwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:52:04.049567 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.049482 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:52:04.049567 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.049551 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "201c55d1-b1fb-4a6c-a4a6-504a056057a5" (UID: "201c55d1-b1fb-4a6c-a4a6-504a056057a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:52:04.148177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148131 2519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148168 2519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148179 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6hwv\" (UniqueName: \"kubernetes.io/projected/201c55d1-b1fb-4a6c-a4a6-504a056057a5-kube-api-access-r6hwv\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148177 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148189 2519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148444 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148197 2519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148444 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148205 2519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/201c55d1-b1fb-4a6c-a4a6-504a056057a5-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.148444 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.148213 2519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/201c55d1-b1fb-4a6c-a4a6-504a056057a5-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:52:04.834625 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.834587 2519 generic.go:356] "Generic (PLEG): container finished" podID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" containerID="587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a" exitCode=0 Sep 11 13:52:04.835097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.834640 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d68cc55c-pjjdk" Sep 11 13:52:04.835097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.834667 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d68cc55c-pjjdk" event={"ID":"201c55d1-b1fb-4a6c-a4a6-504a056057a5","Type":"ContainerDied","Data":"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a"} Sep 11 13:52:04.835097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.834720 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d68cc55c-pjjdk" event={"ID":"201c55d1-b1fb-4a6c-a4a6-504a056057a5","Type":"ContainerDied","Data":"2f6c2874d803ca9e08aa36ae75ce6a04e568189cca083e2cc57a61033a896f59"} Sep 11 13:52:04.835097 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.834740 2519 scope.go:117] "RemoveContainer" containerID="587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a" Sep 11 13:52:04.842869 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.842851 2519 scope.go:117] "RemoveContainer" containerID="587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a" Sep 11 13:52:04.843144 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:52:04.843126 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a\": container with ID starting with 587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a not found: ID does not exist" containerID="587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a" Sep 11 13:52:04.843198 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.843154 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a"} err="failed to get container status \"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a\": rpc error: code = NotFound desc = could not find container \"587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a\": container with ID starting with 587c1c4c5fd58daaca5640e8a8804b6c75bcd65358e50cefe42e8fc1be29849a not found: ID does not exist" Sep 11 13:52:04.850879 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.850855 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:52:04.854500 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:04.854477 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66d68cc55c-pjjdk"] Sep 11 13:52:06.689633 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:06.689601 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" path="/var/lib/kubelet/pods/201c55d1-b1fb-4a6c-a4a6-504a056057a5/volumes" Sep 11 13:52:08.723990 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:08.723960 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:08.824017 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:08.823983 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:17.981150 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:17.981118 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:18.032904 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:18.032874 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:30.539700 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:30.539665 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:31.382426 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:31.382386 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:34.145595 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:34.145550 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:34.245482 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:34.245447 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:35.813823 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:35.813789 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:35.934630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:35.934594 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:36.471065 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:36.471031 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:36.565918 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:36.565884 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:38.864835 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:38.864800 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:39.087338 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:39.087306 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:40.981161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:40.981122 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:41.074315 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:41.074278 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:46.470280 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.470244 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:46.772366 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772279 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gzmsr"] Sep 11 13:52:46.772649 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772636 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" containerName="console" Sep 11 13:52:46.772710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772651 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" containerName="console" Sep 11 13:52:46.772710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772664 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" containerName="console" Sep 11 13:52:46.772710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772670 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" containerName="console" Sep 11 13:52:46.772819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772713 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d3c288-1ccc-4647-8a51-f18e6b6c2e34" containerName="console" Sep 11 13:52:46.772819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.772721 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="201c55d1-b1fb-4a6c-a4a6-504a056057a5" containerName="console" Sep 11 13:52:46.775629 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.775610 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:46.777953 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.777930 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wggkv\"" Sep 11 13:52:46.778050 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.777934 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 13:52:46.778256 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.778242 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 13:52:46.786109 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.786088 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gzmsr"] Sep 11 13:52:46.834652 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.834620 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:46.919116 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.919082 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:46.919286 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:46.919128 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrtk\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-kube-api-access-xmrtk\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.020556 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.020526 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrtk\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-kube-api-access-xmrtk\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.020744 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.020636 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.032528 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.032464 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.032651 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.032538 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrtk\" (UniqueName: \"kubernetes.io/projected/9182b1b6-7238-4af4-8739-6448773dd9f6-kube-api-access-xmrtk\") pod \"cert-manager-55b4d954d5-gzmsr\" (UID: \"9182b1b6-7238-4af4-8739-6448773dd9f6\") " pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.095965 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.095933 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gzmsr" Sep 11 13:52:47.367785 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:47.367756 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gzmsr"] Sep 11 13:52:47.370342 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:52:47.370304 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9182b1b6_7238_4af4_8739_6448773dd9f6.slice/crio-e17f78bd7b838cbeec83215608435c83dfe0493f763d593d9e5372172aa4376d WatchSource:0}: Error finding container e17f78bd7b838cbeec83215608435c83dfe0493f763d593d9e5372172aa4376d: Status 404 returned error can't find the container with id e17f78bd7b838cbeec83215608435c83dfe0493f763d593d9e5372172aa4376d Sep 11 13:52:48.010892 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:48.010854 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gzmsr" event={"ID":"9182b1b6-7238-4af4-8739-6448773dd9f6","Type":"ContainerStarted","Data":"e17f78bd7b838cbeec83215608435c83dfe0493f763d593d9e5372172aa4376d"} Sep 11 13:52:51.026890 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:51.026795 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gzmsr" event={"ID":"9182b1b6-7238-4af4-8739-6448773dd9f6","Type":"ContainerStarted","Data":"562a662497f11f6dc643b2c0defa2407fc3ce69efd602183bf014c23d1067ca0"} Sep 11 13:52:51.043147 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:51.043097 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gzmsr" podStartSLOduration=1.78997893 podStartE2EDuration="5.043082148s" totalImagesPullingTime="3.253103218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:51.041790728 +0000 UTC m=+238.796105620" watchObservedRunningTime="2026-09-11 13:52:51.043082148 +0000 UTC m=+238.797397038" Sep 11 13:52:51.141357 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:51.141324 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:51.242256 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:51.242225 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:52.888229 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.888146 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w"] Sep 11 13:52:52.892891 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.892870 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:52.894844 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.894824 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:52:52.895340 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.895322 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dnpfg\"" Sep 11 13:52:52.895419 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.895356 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 13:52:52.898716 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.898692 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w"] Sep 11 13:52:52.968671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.968635 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcpp\" (UniqueName: \"kubernetes.io/projected/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-kube-api-access-kzcpp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:52.968843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:52.968696 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.069471 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.069436 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcpp\" (UniqueName: \"kubernetes.io/projected/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-kube-api-access-kzcpp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.069709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.069486 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.069877 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.069859 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.078520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.078497 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcpp\" (UniqueName: \"kubernetes.io/projected/a78b44e4-e833-4f97-b0f1-ebdfd83e8979-kube-api-access-kzcpp\") pod \"openshift-lws-operator-fd8ccff4c-zdt7w\" (UID: \"a78b44e4-e833-4f97-b0f1-ebdfd83e8979\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.202709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.202616 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" Sep 11 13:52:53.473427 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:53.473403 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w"] Sep 11 13:52:53.475464 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:52:53.475438 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda78b44e4_e833_4f97_b0f1_ebdfd83e8979.slice/crio-6c03637c6eeae4b3f6ecb27dd95ddf0a70d4e62ab957690224aa6a7e20636936 WatchSource:0}: Error finding container 6c03637c6eeae4b3f6ecb27dd95ddf0a70d4e62ab957690224aa6a7e20636936: Status 404 returned error can't find the container with id 6c03637c6eeae4b3f6ecb27dd95ddf0a70d4e62ab957690224aa6a7e20636936 Sep 11 13:52:54.040461 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:54.040422 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" event={"ID":"a78b44e4-e833-4f97-b0f1-ebdfd83e8979","Type":"ContainerStarted","Data":"6c03637c6eeae4b3f6ecb27dd95ddf0a70d4e62ab957690224aa6a7e20636936"} Sep 11 13:52:57.054643 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:57.054607 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" event={"ID":"a78b44e4-e833-4f97-b0f1-ebdfd83e8979","Type":"ContainerStarted","Data":"36f440576322d5d7c016a88428c9e391c53039d5fa3f77ca32ea24d095a34cfd"} Sep 11 13:52:57.071457 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:57.071404 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zdt7w" podStartSLOduration=2.544314054 podStartE2EDuration="5.071388708s" totalImagesPullingTime="2.527074654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:57.069869266 +0000 UTC m=+244.824184242" watchObservedRunningTime="2026-09-11 13:52:57.071388708 +0000 UTC m=+244.825703608" Sep 11 13:52:57.732143 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:57.732106 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:57.831599 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:57.831551 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:52:59.700449 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:52:59.700415 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:06.252485 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:06.252447 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:06.352438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:06.352407 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:08.848929 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:08.848893 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:08.948667 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:08.948638 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:15.348022 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:15.347940 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:15.446889 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:15.446848 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.012156 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.012122 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.083473 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.083441 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s"] Sep 11 13:53:16.086822 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.086806 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.089931 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.089907 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ql4wz\"" Sep 11 13:53:16.090843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.090824 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 13:53:16.090973 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.090955 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 13:53:16.093864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.093844 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 13:53:16.104500 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.104477 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s"] Sep 11 13:53:16.168274 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.168241 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-metrics-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.168458 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.168286 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48124b0a-fc95-4979-8a6c-50bff066a934-manager-config\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.168458 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.168316 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.168458 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.168428 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77kb\" (UniqueName: \"kubernetes.io/projected/48124b0a-fc95-4979-8a6c-50bff066a934-kube-api-access-s77kb\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.179806 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.179773 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.269854 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.269744 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-metrics-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.269854 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.269818 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48124b0a-fc95-4979-8a6c-50bff066a934-manager-config\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.269854 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.269844 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.270135 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.269900 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77kb\" (UniqueName: \"kubernetes.io/projected/48124b0a-fc95-4979-8a6c-50bff066a934-kube-api-access-s77kb\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.270506 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.270475 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/48124b0a-fc95-4979-8a6c-50bff066a934-manager-config\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.272316 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.272294 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-metrics-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.272408 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.272348 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48124b0a-fc95-4979-8a6c-50bff066a934-cert\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.283784 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.283759 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77kb\" (UniqueName: \"kubernetes.io/projected/48124b0a-fc95-4979-8a6c-50bff066a934-kube-api-access-s77kb\") pod \"lws-controller-manager-8546dd6968-ng94s\" (UID: \"48124b0a-fc95-4979-8a6c-50bff066a934\") " pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.353820 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.353786 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.396492 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.396458 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:16.457355 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.457272 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.525136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.525051 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr"] Sep 11 13:53:16.528792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.528771 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.538942 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.538915 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 13:53:16.539052 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.539026 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 13:53:16.539052 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.539036 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 13:53:16.539153 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.539140 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8wssf\"" Sep 11 13:53:16.543866 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.543851 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 13:53:16.561937 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.561913 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr"] Sep 11 13:53:16.670703 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.670669 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s"] Sep 11 13:53:16.672633 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:53:16.672602 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48124b0a_fc95_4979_8a6c_50bff066a934.slice/crio-7cfd6b1c2fba8c8e4196c9147940c2b20efba49ab644697716f2adc8f26c3f62 WatchSource:0}: Error finding container 7cfd6b1c2fba8c8e4196c9147940c2b20efba49ab644697716f2adc8f26c3f62: Status 404 returned error can't find the container with id 7cfd6b1c2fba8c8e4196c9147940c2b20efba49ab644697716f2adc8f26c3f62 Sep 11 13:53:16.673526 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.673507 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-manifests\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.673640 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.673545 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22v5j\" (UniqueName: \"kubernetes.io/projected/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-kube-api-access-22v5j\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.673692 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.673633 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-tmp\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.673692 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.673680 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.673827 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.673809 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.756619 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.756544 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.774801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.774772 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.774801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.774804 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-manifests\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.774979 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.774830 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22v5j\" (UniqueName: \"kubernetes.io/projected/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-kube-api-access-22v5j\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.774979 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.774860 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-tmp\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.774979 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.774887 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.775274 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.775225 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-manifests\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.777197 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.777173 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-tmp\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.777352 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.777334 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.777449 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.777433 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-webhook-cert\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.783983 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.783956 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22v5j\" (UniqueName: \"kubernetes.io/projected/cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca-kube-api-access-22v5j\") pod \"opendatahub-operator-controller-manager-85cf544d9c-gdtgr\" (UID: \"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca\") " pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.839125 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.839093 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:16.891767 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.891656 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:16.991502 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:16.991467 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:17.100399 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.100372 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:17.132376 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.132347 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr"] Sep 11 13:53:17.143813 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.143786 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" event={"ID":"48124b0a-fc95-4979-8a6c-50bff066a934","Type":"ContainerStarted","Data":"7cfd6b1c2fba8c8e4196c9147940c2b20efba49ab644697716f2adc8f26c3f62"} Sep 11 13:53:17.163717 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:53:17.163688 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc07ec83_33bb_4aa9_ac49_98c3c3b7d6ca.slice/crio-0607ebdfe1262ae56c9bc3592f4dd415c46a7c59c89efef8e991971c3fd063f8 WatchSource:0}: Error finding container 0607ebdfe1262ae56c9bc3592f4dd415c46a7c59c89efef8e991971c3fd063f8: Status 404 returned error can't find the container with id 0607ebdfe1262ae56c9bc3592f4dd415c46a7c59c89efef8e991971c3fd063f8 Sep 11 13:53:17.228398 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.228367 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:17.328941 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.328908 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:17.429937 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:17.429851 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:18.155397 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:18.155350 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" event={"ID":"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca","Type":"ContainerStarted","Data":"0607ebdfe1262ae56c9bc3592f4dd415c46a7c59c89efef8e991971c3fd063f8"} Sep 11 13:53:21.171083 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:21.171045 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" event={"ID":"48124b0a-fc95-4979-8a6c-50bff066a934","Type":"ContainerStarted","Data":"07518c3bb57d6c04604b7df669c90f775ec4870ea47b571d9fecdfee2b83e2a9"} Sep 11 13:53:21.171530 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:21.171108 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:21.172555 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:21.172533 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" event={"ID":"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca","Type":"ContainerStarted","Data":"f7f70043e15bf8147db057a68df6ac4d3b170e3f25a338ecf926713a487ada02"} Sep 11 13:53:21.190444 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:21.190393 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" podStartSLOduration=1.6026738969999998 podStartE2EDuration="5.190378522s" totalImagesPullingTime="3.587704625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:21.189156911 +0000 UTC m=+268.943471795" watchObservedRunningTime="2026-09-11 13:53:21.190378522 +0000 UTC m=+268.944693410" Sep 11 13:53:22.179763 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:22.179730 2519 generic.go:356] "Generic (PLEG): container finished" podID="cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca" containerID="f7f70043e15bf8147db057a68df6ac4d3b170e3f25a338ecf926713a487ada02" exitCode=0 Sep 11 13:53:22.180189 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:22.179827 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" event={"ID":"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca","Type":"ContainerDied","Data":"f7f70043e15bf8147db057a68df6ac4d3b170e3f25a338ecf926713a487ada02"} Sep 11 13:53:23.192449 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:23.192415 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" event={"ID":"cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca","Type":"ContainerStarted","Data":"5986f02d28607a015ba669a334adf51e166e3836c8ca116225ddeda1061717a3"} Sep 11 13:53:23.192843 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:23.192499 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:23.213077 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:23.213031 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" podStartSLOduration=2.6508048239999997 podStartE2EDuration="7.213015556s" totalImagesPullingTime="3.489873414s" totalInitContainerRuntime="1.072337318s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:23.211076971 +0000 UTC m=+270.965391875" watchObservedRunningTime="2026-09-11 13:53:23.213015556 +0000 UTC m=+270.967330446" Sep 11 13:53:32.182647 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:32.182614 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8546dd6968-ng94s" Sep 11 13:53:34.199611 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:34.199556 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85cf544d9c-gdtgr" Sep 11 13:53:36.248723 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.248683 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:36.363364 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.363330 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:36.648161 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.648127 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:36.662286 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.662257 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-846c6f95fb-254hr"] Sep 11 13:53:36.671145 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.671121 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.673552 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.673530 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 13:53:36.673661 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.673539 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 13:53:36.673729 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.673710 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9jw4z\"" Sep 11 13:53:36.674470 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.674451 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 13:53:36.674553 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.674455 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 13:53:36.691595 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.691555 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-846c6f95fb-254hr"] Sep 11 13:53:36.762132 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.762088 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b08e90-2538-4641-bc75-26a575cc8cfe-tls-certs\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.762132 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.762132 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b08e90-2538-4641-bc75-26a575cc8cfe-tmp\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.762341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.762151 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5nn\" (UniqueName: \"kubernetes.io/projected/38b08e90-2538-4641-bc75-26a575cc8cfe-kube-api-access-zk5nn\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.764657 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.764631 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:36.863054 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.863020 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b08e90-2538-4641-bc75-26a575cc8cfe-tls-certs\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.863054 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.863056 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b08e90-2538-4641-bc75-26a575cc8cfe-tmp\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.863302 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.863078 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5nn\" (UniqueName: \"kubernetes.io/projected/38b08e90-2538-4641-bc75-26a575cc8cfe-kube-api-access-zk5nn\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.865386 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.865361 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b08e90-2538-4641-bc75-26a575cc8cfe-tmp\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.865646 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.865628 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38b08e90-2538-4641-bc75-26a575cc8cfe-tls-certs\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.884747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.884715 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5nn\" (UniqueName: \"kubernetes.io/projected/38b08e90-2538-4641-bc75-26a575cc8cfe-kube-api-access-zk5nn\") pod \"kube-auth-proxy-846c6f95fb-254hr\" (UID: \"38b08e90-2538-4641-bc75-26a575cc8cfe\") " pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:36.980950 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:36.980861 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" Sep 11 13:53:37.251886 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:37.251858 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-846c6f95fb-254hr"] Sep 11 13:53:37.254203 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:53:37.254176 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b08e90_2538_4641_bc75_26a575cc8cfe.slice/crio-928b1b2c523eed7d5f5ffceaa15fc8644f51e075ec4d03e32912d7059d5de99a WatchSource:0}: Error finding container 928b1b2c523eed7d5f5ffceaa15fc8644f51e075ec4d03e32912d7059d5de99a: Status 404 returned error can't find the container with id 928b1b2c523eed7d5f5ffceaa15fc8644f51e075ec4d03e32912d7059d5de99a Sep 11 13:53:37.999268 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:37.999200 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:38.100946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:38.100860 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:38.265698 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:38.265592 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" event={"ID":"38b08e90-2538-4641-bc75-26a575cc8cfe","Type":"ContainerStarted","Data":"928b1b2c523eed7d5f5ffceaa15fc8644f51e075ec4d03e32912d7059d5de99a"} Sep 11 13:53:38.972747 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:38.972707 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:40.278367 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:40.278280 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" event={"ID":"38b08e90-2538-4641-bc75-26a575cc8cfe","Type":"ContainerStarted","Data":"1ccbebb4c1d39fd9ef0f09b5c80302514c18fe0fb108d363266c412ba5f88a94"} Sep 11 13:53:40.294353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:40.294299 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-846c6f95fb-254hr" podStartSLOduration=1.665424392 podStartE2EDuration="4.29428186s" totalImagesPullingTime="2.628857468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:40.293048877 +0000 UTC m=+288.047363772" watchObservedRunningTime="2026-09-11 13:53:40.29428186 +0000 UTC m=+288.048596751" Sep 11 13:53:45.358166 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:45.358105 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:46.694128 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:46.694099 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:47.142819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:47.142787 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:47.241805 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:47.241774 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:47.338630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:47.338567 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:47.681738 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:47.681703 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:47.804387 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:47.804356 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:52.633282 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:52.633250 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:53:52.635447 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:52.635418 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:53:52.637306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:52.637290 2519 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 13:53:56.713328 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:56.713284 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:56.841836 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:56.841802 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:56.933101 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:56.933071 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:57.542092 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:57.542058 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:58.089712 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:58.089675 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:58.191698 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:58.191667 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:58.291545 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:53:58.291514 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:48.302071 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:48.301974 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:48.389231 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:48.389195 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:50.462768 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.462731 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d76db8fd-xtdpn"] Sep 11 13:54:50.466067 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.466046 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.476741 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.476719 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d76db8fd-xtdpn"] Sep 11 13:54:50.527483 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.527455 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:50.634765 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634734 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnm5\" (UniqueName: \"kubernetes.io/projected/1e382750-176b-4fe3-ad57-b919d4281589-kube-api-access-wjnm5\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.634941 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634777 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.634941 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634846 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-console-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.634941 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634908 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-trusted-ca-bundle\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.635045 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634947 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-service-ca\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.635045 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.634978 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-oauth-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.635112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.635041 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-oauth-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.652495 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.652471 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:50.724111 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.724045 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:50.736249 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736212 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-oauth-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736255 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnm5\" (UniqueName: \"kubernetes.io/projected/1e382750-176b-4fe3-ad57-b919d4281589-kube-api-access-wjnm5\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736291 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736336 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-console-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736370 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-trusted-ca-bundle\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736408 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-service-ca\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.736724 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.736452 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-oauth-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.737112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.737085 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-oauth-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.737233 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.737198 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-service-ca\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.737311 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.737292 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-trusted-ca-bundle\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.737364 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.737340 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e382750-176b-4fe3-ad57-b919d4281589-console-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.738855 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.738833 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-oauth-config\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.738968 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.738948 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e382750-176b-4fe3-ad57-b919d4281589-console-serving-cert\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.744335 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.744312 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnm5\" (UniqueName: \"kubernetes.io/projected/1e382750-176b-4fe3-ad57-b919d4281589-kube-api-access-wjnm5\") pod \"console-6d76db8fd-xtdpn\" (UID: \"1e382750-176b-4fe3-ad57-b919d4281589\") " pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:50.776392 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:50.776368 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:54:51.048393 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.048371 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d76db8fd-xtdpn"] Sep 11 13:54:51.050594 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:54:51.050545 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e382750_176b_4fe3_ad57_b919d4281589.slice/crio-4886f76f15bd43a3e649eff718113b9de78023ffa2e5f3b16294a53e7136a8a4 WatchSource:0}: Error finding container 4886f76f15bd43a3e649eff718113b9de78023ffa2e5f3b16294a53e7136a8a4: Status 404 returned error can't find the container with id 4886f76f15bd43a3e649eff718113b9de78023ffa2e5f3b16294a53e7136a8a4 Sep 11 13:54:51.257812 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.257768 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:51.324069 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.323985 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:51.619995 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.619963 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d76db8fd-xtdpn" event={"ID":"1e382750-176b-4fe3-ad57-b919d4281589","Type":"ContainerStarted","Data":"e98f42992fe6572f7c3b81b9e04a1bacaa9069b29293dd6fce8a36ab493c85fa"} Sep 11 13:54:51.620361 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.620011 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d76db8fd-xtdpn" event={"ID":"1e382750-176b-4fe3-ad57-b919d4281589","Type":"ContainerStarted","Data":"4886f76f15bd43a3e649eff718113b9de78023ffa2e5f3b16294a53e7136a8a4"} Sep 11 13:54:51.640358 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:51.640306 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d76db8fd-xtdpn" podStartSLOduration=1.6402914069999999 podStartE2EDuration="1.640291407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:51.638614465 +0000 UTC m=+359.392929358" watchObservedRunningTime="2026-09-11 13:54:51.640291407 +0000 UTC m=+359.394606298" Sep 11 13:54:59.907073 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:54:59.907044 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:00.777138 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:00.777097 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:55:00.777378 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:00.777164 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:55:00.782052 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:00.782027 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:55:01.673698 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:01.673671 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d76db8fd-xtdpn" Sep 11 13:55:01.730750 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:01.730711 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:55:01.854450 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:01.854421 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:01.954885 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:01.954774 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:02.075123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:02.075094 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:02.239313 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:02.239217 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:03.019814 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:03.019776 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:03.828869 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:03.828827 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:06.937311 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:06.937261 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:07.070488 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:07.070452 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:07.151640 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:07.151601 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:07.441509 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:07.441467 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:08.488836 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:08.488807 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:08.589633 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:08.589604 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:10.048029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:10.047998 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:10.145048 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:10.144996 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:12.215430 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:12.215395 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:13.704298 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:13.704262 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:14.707463 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:14.707427 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:14.941227 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:14.941195 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.430402 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:17.430352 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.493618 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:17.493549 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.820795 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:17.820718 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.933353 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:17.933320 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:18.219101 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:18.219069 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:18.320176 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:18.320146 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:25.902041 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.902007 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:25.906406 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.906370 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:25.908643 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.908616 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-882pd\"" Sep 11 13:55:25.908643 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.908630 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 13:55:25.908828 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.908619 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 13:55:25.920936 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:25.920907 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:26.055408 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.055370 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.055611 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.055488 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zcj\" (UniqueName: \"kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.156329 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.156238 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.156509 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.156336 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zcj\" (UniqueName: \"kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.156767 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.156739 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.177323 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.177291 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zcj\" (UniqueName: \"kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rvsg8\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.217079 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.217047 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:26.266798 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.266761 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:26.369861 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.369828 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:26.510611 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.510512 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:26.514051 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:55:26.514023 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d17fa67_b19a_4d0b_b803_aa8af31707b2.slice/crio-88ca92f4b088a62ed21c698bf01085997929a82e06b083ee8b8f9ab987bc1d2e WatchSource:0}: Error finding container 88ca92f4b088a62ed21c698bf01085997929a82e06b083ee8b8f9ab987bc1d2e: Status 404 returned error can't find the container with id 88ca92f4b088a62ed21c698bf01085997929a82e06b083ee8b8f9ab987bc1d2e Sep 11 13:55:26.516820 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.516800 2519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 13:55:26.750440 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.750398 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-854bf86c8f-dtxfz" podUID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" containerName="console" containerID="cri-o://dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2" gracePeriod=15 Sep 11 13:55:26.787751 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.787725 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" event={"ID":"0d17fa67-b19a-4d0b-b803-aa8af31707b2","Type":"ContainerStarted","Data":"88ca92f4b088a62ed21c698bf01085997929a82e06b083ee8b8f9ab987bc1d2e"} Sep 11 13:55:26.987179 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:26.987155 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:55:27.062914 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062841 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.062914 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062904 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063096 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062921 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063096 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062938 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d589v\" (UniqueName: \"kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063096 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062973 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063096 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.062995 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063096 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.063020 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config\") pod \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\" (UID: \"b2224931-7a69-4bc0-b6fe-0a93a0554e9d\") " Sep 11 13:55:27.063440 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.063290 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:27.063440 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.063411 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config" (OuterVolumeSpecName: "console-config") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:27.063440 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.063402 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:27.063716 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.063507 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:27.065260 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.065236 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:55:27.065339 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.065277 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v" (OuterVolumeSpecName: "kube-api-access-d589v") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "kube-api-access-d589v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:55:27.065339 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.065321 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2224931-7a69-4bc0-b6fe-0a93a0554e9d" (UID: "b2224931-7a69-4bc0-b6fe-0a93a0554e9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:55:27.164169 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164131 2519 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-oauth-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164169 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164169 2519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-trusted-ca-bundle\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164181 2519 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-serving-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164193 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d589v\" (UniqueName: \"kubernetes.io/projected/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-kube-api-access-d589v\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164205 2519 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164218 2519 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.164425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.164228 2519 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2224931-7a69-4bc0-b6fe-0a93a0554e9d-console-oauth-config\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:27.796247 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.796194 2519 generic.go:356] "Generic (PLEG): container finished" podID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" containerID="dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2" exitCode=0 Sep 11 13:55:27.796437 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.796259 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854bf86c8f-dtxfz" Sep 11 13:55:27.796437 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.796310 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854bf86c8f-dtxfz" event={"ID":"b2224931-7a69-4bc0-b6fe-0a93a0554e9d","Type":"ContainerDied","Data":"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2"} Sep 11 13:55:27.796437 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.796367 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854bf86c8f-dtxfz" event={"ID":"b2224931-7a69-4bc0-b6fe-0a93a0554e9d","Type":"ContainerDied","Data":"4a6bf5c0bde70c1bc080fe391d449baf34b04a402bc1fb48a88a1872705b49e1"} Sep 11 13:55:27.796437 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.796397 2519 scope.go:117] "RemoveContainer" containerID="dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2" Sep 11 13:55:27.806966 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.806947 2519 scope.go:117] "RemoveContainer" containerID="dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2" Sep 11 13:55:27.807287 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:55:27.807261 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2\": container with ID starting with dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2 not found: ID does not exist" containerID="dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2" Sep 11 13:55:27.807382 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.807299 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2"} err="failed to get container status \"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2\": rpc error: code = NotFound desc = could not find container \"dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2\": container with ID starting with dbc2607433867bfba59a6fe6c0fb979a678b20426caf5b6f751e29e2e2c081e2 not found: ID does not exist" Sep 11 13:55:27.823607 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.823549 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:55:27.830329 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:27.830304 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854bf86c8f-dtxfz"] Sep 11 13:55:28.694524 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:28.694479 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" path="/var/lib/kubelet/pods/b2224931-7a69-4bc0-b6fe-0a93a0554e9d/volumes" Sep 11 13:55:28.703458 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:28.703427 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:28.803775 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:28.803734 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:30.816232 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:30.816193 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" event={"ID":"0d17fa67-b19a-4d0b-b803-aa8af31707b2","Type":"ContainerStarted","Data":"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965"} Sep 11 13:55:30.816655 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:30.816330 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:30.835804 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:30.835757 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" podStartSLOduration=2.120218369 podStartE2EDuration="5.835739254s" totalImagesPullingTime="3.715520885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:30.835003993 +0000 UTC m=+398.589318910" watchObservedRunningTime="2026-09-11 13:55:30.835739254 +0000 UTC m=+398.590054149" Sep 11 13:55:31.103461 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:31.103432 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:33.450771 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:33.450740 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:33.550899 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:33.550870 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:37.943497 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:37.943465 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:37.943946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:37.943659 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" podUID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" containerName="manager" containerID="cri-o://45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965" gracePeriod=10 Sep 11 13:55:37.945629 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:37.945606 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:38.191563 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.191535 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:38.371398 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.371359 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume\") pod \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " Sep 11 13:55:38.371595 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.371501 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2zcj\" (UniqueName: \"kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj\") pod \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\" (UID: \"0d17fa67-b19a-4d0b-b803-aa8af31707b2\") " Sep 11 13:55:38.371879 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.371845 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0d17fa67-b19a-4d0b-b803-aa8af31707b2" (UID: "0d17fa67-b19a-4d0b-b803-aa8af31707b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:55:38.373637 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.373609 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj" (OuterVolumeSpecName: "kube-api-access-z2zcj") pod "0d17fa67-b19a-4d0b-b803-aa8af31707b2" (UID: "0d17fa67-b19a-4d0b-b803-aa8af31707b2"). InnerVolumeSpecName "kube-api-access-z2zcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:55:38.472862 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.472828 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zcj\" (UniqueName: \"kubernetes.io/projected/0d17fa67-b19a-4d0b-b803-aa8af31707b2-kube-api-access-z2zcj\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:38.472862 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.472855 2519 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d17fa67-b19a-4d0b-b803-aa8af31707b2-extensions-socket-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:55:38.649046 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.648943 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:38.752283 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.752251 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:38.855056 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.855016 2519 generic.go:356] "Generic (PLEG): container finished" podID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" containerID="45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965" exitCode=0 Sep 11 13:55:38.855056 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.855058 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" Sep 11 13:55:38.855310 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.855097 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" event={"ID":"0d17fa67-b19a-4d0b-b803-aa8af31707b2","Type":"ContainerDied","Data":"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965"} Sep 11 13:55:38.855310 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.855152 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8" event={"ID":"0d17fa67-b19a-4d0b-b803-aa8af31707b2","Type":"ContainerDied","Data":"88ca92f4b088a62ed21c698bf01085997929a82e06b083ee8b8f9ab987bc1d2e"} Sep 11 13:55:38.855310 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.855182 2519 scope.go:117] "RemoveContainer" containerID="45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965" Sep 11 13:55:38.863889 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.863874 2519 scope.go:117] "RemoveContainer" containerID="45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965" Sep 11 13:55:38.864143 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:55:38.864124 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965\": container with ID starting with 45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965 not found: ID does not exist" containerID="45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965" Sep 11 13:55:38.864193 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.864151 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965"} err="failed to get container status \"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965\": rpc error: code = NotFound desc = could not find container \"45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965\": container with ID starting with 45ee250a288f5bc840cccc4a8acf9b44d644106e1ee642dc7b11a7935ef85965 not found: ID does not exist" Sep 11 13:55:38.881136 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.881114 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:38.886921 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:38.886902 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rvsg8"] Sep 11 13:55:39.045007 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:39.044920 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:39.145397 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:39.145360 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:39.261766 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:39.261731 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:39.370469 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:39.370441 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:40.690104 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:40.690064 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" path="/var/lib/kubelet/pods/0d17fa67-b19a-4d0b-b803-aa8af31707b2/volumes" Sep 11 13:55:48.041630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:48.041594 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:48.142974 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:48.142941 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:50.498797 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:50.498761 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:50.557168 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:50.557128 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:54.355053 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:54.355012 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:54.502801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:54.502770 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:55.245245 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:55.245209 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:55.348864 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:55.348830 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:56.422792 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.422758 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:55:56.423171 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423134 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" containerName="manager" Sep 11 13:55:56.423171 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423145 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" containerName="manager" Sep 11 13:55:56.423240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423174 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" containerName="console" Sep 11 13:55:56.423240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423180 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" containerName="console" Sep 11 13:55:56.423240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423231 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d17fa67-b19a-4d0b-b803-aa8af31707b2" containerName="manager" Sep 11 13:55:56.423333 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.423243 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2224931-7a69-4bc0-b6fe-0a93a0554e9d" containerName="console" Sep 11 13:55:56.426584 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.426545 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:56.427462 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.427443 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.429373 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.429354 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 13:55:56.429373 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.429356 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 13:55:56.429547 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.429432 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 13:55:56.429937 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.429905 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xzr8s\"" Sep 11 13:55:56.436125 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.436103 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:55:56.525616 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.525586 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:55:56.528269 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.528235 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:56.542507 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.542471 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkcs\" (UniqueName: \"kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.542692 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.542633 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.628624 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.628568 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:56.644029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.643999 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkcs\" (UniqueName: \"kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.644266 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.644074 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.644648 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.644631 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.651728 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.651710 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkcs\" (UniqueName: \"kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs\") pod \"limitador-limitador-8588749977-f85p6\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:56.738545 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:56.738465 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:57.013933 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.013902 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:55:57.015945 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:55:57.015915 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f4276f_90a6_45c5_af4b_c5cc48f07b04.slice/crio-f8d3e1fdaf7d221d566f1f7ccbfac3a671aa422d0bbd743cf503ad30e5170496 WatchSource:0}: Error finding container f8d3e1fdaf7d221d566f1f7ccbfac3a671aa422d0bbd743cf503ad30e5170496: Status 404 returned error can't find the container with id f8d3e1fdaf7d221d566f1f7ccbfac3a671aa422d0bbd743cf503ad30e5170496 Sep 11 13:55:57.230295 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.230258 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:55:57.234293 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.234271 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:55:57.236697 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.236681 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kbqgj\"" Sep 11 13:55:57.246957 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.246931 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:55:57.280208 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.280136 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:57.349589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.349549 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtmv\" (UniqueName: \"kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv\") pod \"authorino-7bcfd8f9cd-fvwrp\" (UID: \"939dc1e6-bf2c-42eb-8fb1-909d55378b2f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:55:57.386725 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.386691 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:57.446927 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.446895 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:55:57.449378 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.449362 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:55:57.450667 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.450640 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtmv\" (UniqueName: \"kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv\") pod \"authorino-7bcfd8f9cd-fvwrp\" (UID: \"939dc1e6-bf2c-42eb-8fb1-909d55378b2f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:55:57.454867 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.454844 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:55:57.459402 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.459373 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtmv\" (UniqueName: \"kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv\") pod \"authorino-7bcfd8f9cd-fvwrp\" (UID: \"939dc1e6-bf2c-42eb-8fb1-909d55378b2f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:55:57.543107 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.543026 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:55:57.551327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.551298 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4j2\" (UniqueName: \"kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2\") pod \"authorino-59c974bd48-q26gg\" (UID: \"faf178d8-2a71-4130-a786-4f31c15a5f33\") " pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:55:57.652431 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.652396 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4j2\" (UniqueName: \"kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2\") pod \"authorino-59c974bd48-q26gg\" (UID: \"faf178d8-2a71-4130-a786-4f31c15a5f33\") " pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:55:57.661047 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.661017 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4j2\" (UniqueName: \"kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2\") pod \"authorino-59c974bd48-q26gg\" (UID: \"faf178d8-2a71-4130-a786-4f31c15a5f33\") " pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:55:57.759279 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.759252 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:55:57.812090 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.811430 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:55:57.817276 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:55:57.816953 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod939dc1e6_bf2c_42eb_8fb1_909d55378b2f.slice/crio-36bc61d9e47b88c179dc2be72fe212262c39e7906b5970016197c43d79028b8c WatchSource:0}: Error finding container 36bc61d9e47b88c179dc2be72fe212262c39e7906b5970016197c43d79028b8c: Status 404 returned error can't find the container with id 36bc61d9e47b88c179dc2be72fe212262c39e7906b5970016197c43d79028b8c Sep 11 13:55:57.945783 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.945741 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" event={"ID":"939dc1e6-bf2c-42eb-8fb1-909d55378b2f","Type":"ContainerStarted","Data":"36bc61d9e47b88c179dc2be72fe212262c39e7906b5970016197c43d79028b8c"} Sep 11 13:55:57.947287 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:57.947256 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" event={"ID":"73f4276f-90a6-45c5-af4b-c5cc48f07b04","Type":"ContainerStarted","Data":"f8d3e1fdaf7d221d566f1f7ccbfac3a671aa422d0bbd743cf503ad30e5170496"} Sep 11 13:55:58.027107 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:58.027078 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:55:58.029469 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:55:58.029441 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf178d8_2a71_4130_a786_4f31c15a5f33.slice/crio-e228e8e3edb8d2326e4fcc3f1bb02623e6566077b5e8380308aabe80ada11ae3 WatchSource:0}: Error finding container e228e8e3edb8d2326e4fcc3f1bb02623e6566077b5e8380308aabe80ada11ae3: Status 404 returned error can't find the container with id e228e8e3edb8d2326e4fcc3f1bb02623e6566077b5e8380308aabe80ada11ae3 Sep 11 13:55:58.954856 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:58.954808 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" event={"ID":"73f4276f-90a6-45c5-af4b-c5cc48f07b04","Type":"ContainerStarted","Data":"46910d0f60a77c845f48657ced439d96b982365ba3d4cab440ab8968b7bba0c7"} Sep 11 13:55:58.955324 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:58.954885 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:55:58.956383 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:58.956355 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q26gg" event={"ID":"faf178d8-2a71-4130-a786-4f31c15a5f33","Type":"ContainerStarted","Data":"e228e8e3edb8d2326e4fcc3f1bb02623e6566077b5e8380308aabe80ada11ae3"} Sep 11 13:55:59.962968 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:59.962888 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" event={"ID":"939dc1e6-bf2c-42eb-8fb1-909d55378b2f","Type":"ContainerStarted","Data":"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30"} Sep 11 13:55:59.978993 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:59.978690 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" podStartSLOduration=1.114242662 podStartE2EDuration="2.978670664s" totalImagesPullingTime="1.864428002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:59.976048405 +0000 UTC m=+427.730363299" watchObservedRunningTime="2026-09-11 13:55:59.978670664 +0000 UTC m=+427.732985555" Sep 11 13:55:59.979160 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:55:59.979046 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" podStartSLOduration=2.737182201 podStartE2EDuration="3.979035835s" totalImagesPullingTime="1.241853634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:58.97184763 +0000 UTC m=+426.726162522" watchObservedRunningTime="2026-09-11 13:55:59.979035835 +0000 UTC m=+427.733350725" Sep 11 13:56:02.978970 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:02.978930 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q26gg" event={"ID":"faf178d8-2a71-4130-a786-4f31c15a5f33","Type":"ContainerStarted","Data":"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c"} Sep 11 13:56:02.994085 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:02.994036 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-q26gg" podStartSLOduration=1.984114601 podStartE2EDuration="5.994021778s" totalImagesPullingTime="4.009907177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:02.992000711 +0000 UTC m=+430.746315604" watchObservedRunningTime="2026-09-11 13:56:02.994021778 +0000 UTC m=+430.748336668" Sep 11 13:56:03.019913 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.019878 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:56:03.020068 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.020041 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" podUID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" containerName="authorino" containerID="cri-o://67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30" gracePeriod=30 Sep 11 13:56:03.262199 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.262174 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:56:03.302654 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.302621 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jtmv\" (UniqueName: \"kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv\") pod \"939dc1e6-bf2c-42eb-8fb1-909d55378b2f\" (UID: \"939dc1e6-bf2c-42eb-8fb1-909d55378b2f\") " Sep 11 13:56:03.305015 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.304983 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv" (OuterVolumeSpecName: "kube-api-access-7jtmv") pod "939dc1e6-bf2c-42eb-8fb1-909d55378b2f" (UID: "939dc1e6-bf2c-42eb-8fb1-909d55378b2f"). InnerVolumeSpecName "kube-api-access-7jtmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:03.403302 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.403264 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtmv\" (UniqueName: \"kubernetes.io/projected/939dc1e6-bf2c-42eb-8fb1-909d55378b2f-kube-api-access-7jtmv\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:03.984819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.984779 2519 generic.go:356] "Generic (PLEG): container finished" podID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" containerID="67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30" exitCode=0 Sep 11 13:56:03.984819 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.984821 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" Sep 11 13:56:03.985307 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.984866 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" event={"ID":"939dc1e6-bf2c-42eb-8fb1-909d55378b2f","Type":"ContainerDied","Data":"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30"} Sep 11 13:56:03.985307 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.984928 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fvwrp" event={"ID":"939dc1e6-bf2c-42eb-8fb1-909d55378b2f","Type":"ContainerDied","Data":"36bc61d9e47b88c179dc2be72fe212262c39e7906b5970016197c43d79028b8c"} Sep 11 13:56:03.985307 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.984955 2519 scope.go:117] "RemoveContainer" containerID="67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30" Sep 11 13:56:03.993896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.993882 2519 scope.go:117] "RemoveContainer" containerID="67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30" Sep 11 13:56:03.994145 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:03.994126 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30\": container with ID starting with 67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30 not found: ID does not exist" containerID="67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30" Sep 11 13:56:03.994203 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:03.994153 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30"} err="failed to get container status \"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30\": rpc error: code = NotFound desc = could not find container \"67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30\": container with ID starting with 67676038a14d68eb41b0274df30ebfacf595adc3ea24cfcdb95c5caf78b62a30 not found: ID does not exist" Sep 11 13:56:04.005327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:04.005305 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:56:04.011521 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:04.011502 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fvwrp"] Sep 11 13:56:04.691055 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:04.691020 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" path="/var/lib/kubelet/pods/939dc1e6-bf2c-42eb-8fb1-909d55378b2f/volumes" Sep 11 13:56:09.964555 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:09.964526 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:56:27.628107 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:27.628016 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:56:27.628644 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:27.628287 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" podUID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" containerName="limitador" containerID="cri-o://46910d0f60a77c845f48657ced439d96b982365ba3d4cab440ab8968b7bba0c7" gracePeriod=30 Sep 11 13:56:28.104368 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.104334 2519 generic.go:356] "Generic (PLEG): container finished" podID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" containerID="46910d0f60a77c845f48657ced439d96b982365ba3d4cab440ab8968b7bba0c7" exitCode=0 Sep 11 13:56:28.104502 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.104416 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" event={"ID":"73f4276f-90a6-45c5-af4b-c5cc48f07b04","Type":"ContainerDied","Data":"46910d0f60a77c845f48657ced439d96b982365ba3d4cab440ab8968b7bba0c7"} Sep 11 13:56:28.177240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.177215 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:56:28.327341 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.327260 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file\") pod \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " Sep 11 13:56:28.327488 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.327344 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhkcs\" (UniqueName: \"kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs\") pod \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\" (UID: \"73f4276f-90a6-45c5-af4b-c5cc48f07b04\") " Sep 11 13:56:28.327721 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.327701 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file" (OuterVolumeSpecName: "config-file") pod "73f4276f-90a6-45c5-af4b-c5cc48f07b04" (UID: "73f4276f-90a6-45c5-af4b-c5cc48f07b04"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:56:28.329513 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.329479 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs" (OuterVolumeSpecName: "kube-api-access-vhkcs") pod "73f4276f-90a6-45c5-af4b-c5cc48f07b04" (UID: "73f4276f-90a6-45c5-af4b-c5cc48f07b04"). InnerVolumeSpecName "kube-api-access-vhkcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:28.428897 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.428867 2519 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73f4276f-90a6-45c5-af4b-c5cc48f07b04-config-file\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:28.428897 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:28.428895 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhkcs\" (UniqueName: \"kubernetes.io/projected/73f4276f-90a6-45c5-af4b-c5cc48f07b04-kube-api-access-vhkcs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:29.110189 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:29.110160 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" Sep 11 13:56:29.110624 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:29.110195 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-f85p6" event={"ID":"73f4276f-90a6-45c5-af4b-c5cc48f07b04","Type":"ContainerDied","Data":"f8d3e1fdaf7d221d566f1f7ccbfac3a671aa422d0bbd743cf503ad30e5170496"} Sep 11 13:56:29.110624 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:29.110252 2519 scope.go:117] "RemoveContainer" containerID="46910d0f60a77c845f48657ced439d96b982365ba3d4cab440ab8968b7bba0c7" Sep 11 13:56:29.128401 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:29.128364 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:56:29.130824 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:29.130804 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-f85p6"] Sep 11 13:56:30.690946 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:30.690914 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" path="/var/lib/kubelet/pods/73f4276f-90a6-45c5-af4b-c5cc48f07b04/volumes" Sep 11 13:56:32.967326 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:32.967291 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:33.065833 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.065801 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:33.411175 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411141 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj"] Sep 11 13:56:33.411512 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411498 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" containerName="authorino" Sep 11 13:56:33.411564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411514 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" containerName="authorino" Sep 11 13:56:33.411564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411542 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" containerName="limitador" Sep 11 13:56:33.411564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411548 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" containerName="limitador" Sep 11 13:56:33.411682 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411641 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="939dc1e6-bf2c-42eb-8fb1-909d55378b2f" containerName="authorino" Sep 11 13:56:33.411682 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.411652 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f4276f-90a6-45c5-af4b-c5cc48f07b04" containerName="limitador" Sep 11 13:56:33.417286 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.417261 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.419484 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.419463 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 13:56:33.419660 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.419464 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 13:56:33.419931 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.419911 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 13:56:33.419992 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.419911 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w2twr\"" Sep 11 13:56:33.423354 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.423328 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj"] Sep 11 13:56:33.450331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.450295 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:33.468630 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.468602 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9msb\" (UniqueName: \"kubernetes.io/projected/83fcaf09-6ad9-496d-9bd4-43b240b168c7-kube-api-access-k9msb\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.468761 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.468703 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83fcaf09-6ad9-496d-9bd4-43b240b168c7-data\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.569931 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.569897 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83fcaf09-6ad9-496d-9bd4-43b240b168c7-data\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.570112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.569968 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9msb\" (UniqueName: \"kubernetes.io/projected/83fcaf09-6ad9-496d-9bd4-43b240b168c7-kube-api-access-k9msb\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.570385 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.570356 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83fcaf09-6ad9-496d-9bd4-43b240b168c7-data\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.578768 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.578750 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9msb\" (UniqueName: \"kubernetes.io/projected/83fcaf09-6ad9-496d-9bd4-43b240b168c7-kube-api-access-k9msb\") pod \"postgres-69fb4594cf-mp4sj\" (UID: \"83fcaf09-6ad9-496d-9bd4-43b240b168c7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.729631 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.729522 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:33.998347 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:33.998311 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj"] Sep 11 13:56:34.000740 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:56:34.000704 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fcaf09_6ad9_496d_9bd4_43b240b168c7.slice/crio-49998329d37cb2fa705c83c6142742e303b880c25b00cfb053defd49e583fc14 WatchSource:0}: Error finding container 49998329d37cb2fa705c83c6142742e303b880c25b00cfb053defd49e583fc14: Status 404 returned error can't find the container with id 49998329d37cb2fa705c83c6142742e303b880c25b00cfb053defd49e583fc14 Sep 11 13:56:34.136985 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:34.136950 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" event={"ID":"83fcaf09-6ad9-496d-9bd4-43b240b168c7","Type":"ContainerStarted","Data":"49998329d37cb2fa705c83c6142742e303b880c25b00cfb053defd49e583fc14"} Sep 11 13:56:39.170653 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:39.170616 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" event={"ID":"83fcaf09-6ad9-496d-9bd4-43b240b168c7","Type":"ContainerStarted","Data":"7c03d7a630ab8341422c831e866779b72b9a3a8a5bb22fad5120cc1b43a2adf4"} Sep 11 13:56:39.171149 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:39.170708 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:39.188771 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:39.188721 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" podStartSLOduration=1.479615594 podStartE2EDuration="6.18870505s" totalImagesPullingTime="4.709089456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:39.186916267 +0000 UTC m=+466.941231160" watchObservedRunningTime="2026-09-11 13:56:39.18870505 +0000 UTC m=+466.943019955" Sep 11 13:56:40.535514 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:40.535474 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:40.590896 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:40.590854 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:45.203797 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.203771 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mp4sj" Sep 11 13:56:45.512834 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.512755 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:45.613464 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.613435 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:45.736749 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.736717 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-f8v6c"] Sep 11 13:56:45.739495 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.739470 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:45.748903 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.748881 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-f8v6c"] Sep 11 13:56:45.782979 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.782911 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6tq\" (UniqueName: \"kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq\") pod \"authorino-557f876f69-f8v6c\" (UID: \"32cf52a7-5376-4bfd-bf75-c93608e58825\") " pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:45.879069 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.879032 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-f8v6c"] Sep 11 13:56:45.879319 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:45.879301 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mm6tq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-f8v6c" podUID="32cf52a7-5376-4bfd-bf75-c93608e58825" Sep 11 13:56:45.884281 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.884256 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6tq\" (UniqueName: \"kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq\") pod \"authorino-557f876f69-f8v6c\" (UID: \"32cf52a7-5376-4bfd-bf75-c93608e58825\") " pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:45.898247 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.898217 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6tq\" (UniqueName: \"kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq\") pod \"authorino-557f876f69-f8v6c\" (UID: \"32cf52a7-5376-4bfd-bf75-c93608e58825\") " pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:45.905381 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.905360 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pmkt7"] Sep 11 13:56:45.908234 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.908215 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:45.910240 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.910222 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 13:56:45.917110 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.917087 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pmkt7"] Sep 11 13:56:45.974037 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.974005 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pmkt7"] Sep 11 13:56:45.974287 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:45.974269 2519 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5j8sc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" podUID="63d3d929-c797-48dc-9c61-8d4db149d591" Sep 11 13:56:45.985422 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.985391 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:45.985532 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:45.985426 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8sc\" (UniqueName: \"kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.001838 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.001812 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 13:56:46.004928 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.004904 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.012812 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.012790 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 13:56:46.086113 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.086087 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.086246 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.086123 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8sc\" (UniqueName: \"kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.086339 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.086315 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n86\" (UniqueName: \"kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.086397 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.086384 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.088438 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.088422 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.094064 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.094045 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8sc\" (UniqueName: \"kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc\") pod \"authorino-6d4dfc649f-pmkt7\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.187020 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.186983 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n86\" (UniqueName: \"kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.187215 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.187046 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.189437 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.189412 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.194517 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.194497 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n86\" (UniqueName: \"kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86\") pod \"authorino-7498fdbcf5-nzxgh\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.204026 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.204006 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:46.204315 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.204013 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.211278 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.211260 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:46.214295 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.214278 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:46.288210 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.288181 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm6tq\" (UniqueName: \"kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq\") pod \"32cf52a7-5376-4bfd-bf75-c93608e58825\" (UID: \"32cf52a7-5376-4bfd-bf75-c93608e58825\") " Sep 11 13:56:46.288372 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.288257 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j8sc\" (UniqueName: \"kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc\") pod \"63d3d929-c797-48dc-9c61-8d4db149d591\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " Sep 11 13:56:46.288372 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.288312 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert\") pod \"63d3d929-c797-48dc-9c61-8d4db149d591\" (UID: \"63d3d929-c797-48dc-9c61-8d4db149d591\") " Sep 11 13:56:46.290467 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.290432 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "63d3d929-c797-48dc-9c61-8d4db149d591" (UID: "63d3d929-c797-48dc-9c61-8d4db149d591"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:56:46.290467 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.290453 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc" (OuterVolumeSpecName: "kube-api-access-5j8sc") pod "63d3d929-c797-48dc-9c61-8d4db149d591" (UID: "63d3d929-c797-48dc-9c61-8d4db149d591"). InnerVolumeSpecName "kube-api-access-5j8sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:46.290618 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.290477 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq" (OuterVolumeSpecName: "kube-api-access-mm6tq") pod "32cf52a7-5376-4bfd-bf75-c93608e58825" (UID: "32cf52a7-5376-4bfd-bf75-c93608e58825"). InnerVolumeSpecName "kube-api-access-mm6tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:46.314388 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.314363 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 13:56:46.389194 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.389167 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8sc\" (UniqueName: \"kubernetes.io/projected/63d3d929-c797-48dc-9c61-8d4db149d591-kube-api-access-5j8sc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:46.389194 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.389195 2519 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63d3d929-c797-48dc-9c61-8d4db149d591-tls-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:46.389360 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.389206 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6tq\" (UniqueName: \"kubernetes.io/projected/32cf52a7-5376-4bfd-bf75-c93608e58825-kube-api-access-mm6tq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:46.581407 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:46.581373 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 13:56:46.584630 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:56:46.584604 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod257cd4ee_2b2d_4191_a17d_1fae6d689d62.slice/crio-5b9fa07a037c63fc18cfbbaaa500a2f3b22c8f479da01cc2eeed3aaf5ca0024d WatchSource:0}: Error finding container 5b9fa07a037c63fc18cfbbaaa500a2f3b22c8f479da01cc2eeed3aaf5ca0024d: Status 404 returned error can't find the container with id 5b9fa07a037c63fc18cfbbaaa500a2f3b22c8f479da01cc2eeed3aaf5ca0024d Sep 11 13:56:47.212520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.212414 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" event={"ID":"257cd4ee-2b2d-4191-a17d-1fae6d689d62","Type":"ContainerStarted","Data":"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b"} Sep 11 13:56:47.212520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.212445 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-f8v6c" Sep 11 13:56:47.212520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.212462 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" event={"ID":"257cd4ee-2b2d-4191-a17d-1fae6d689d62","Type":"ContainerStarted","Data":"5b9fa07a037c63fc18cfbbaaa500a2f3b22c8f479da01cc2eeed3aaf5ca0024d"} Sep 11 13:56:47.212520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.212434 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pmkt7" Sep 11 13:56:47.229357 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.229311 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" podStartSLOduration=1.694560784 podStartE2EDuration="2.22929326s" totalImagesPullingTime="534.732476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:47.227282214 +0000 UTC m=+474.981597106" watchObservedRunningTime="2026-09-11 13:56:47.22929326 +0000 UTC m=+474.983608151" Sep 11 13:56:47.253606 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.253560 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pmkt7"] Sep 11 13:56:47.260391 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.260362 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:56:47.260616 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.260551 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-q26gg" podUID="faf178d8-2a71-4130-a786-4f31c15a5f33" containerName="authorino" containerID="cri-o://23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c" gracePeriod=30 Sep 11 13:56:47.270812 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.269154 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pmkt7"] Sep 11 13:56:47.301407 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.299294 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-f8v6c"] Sep 11 13:56:47.302941 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.302825 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-f8v6c"] Sep 11 13:56:47.494306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.494282 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:56:47.600277 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.600247 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4j2\" (UniqueName: \"kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2\") pod \"faf178d8-2a71-4130-a786-4f31c15a5f33\" (UID: \"faf178d8-2a71-4130-a786-4f31c15a5f33\") " Sep 11 13:56:47.602269 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.602240 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2" (OuterVolumeSpecName: "kube-api-access-wf4j2") pod "faf178d8-2a71-4130-a786-4f31c15a5f33" (UID: "faf178d8-2a71-4130-a786-4f31c15a5f33"). InnerVolumeSpecName "kube-api-access-wf4j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:47.701166 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:47.701083 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4j2\" (UniqueName: \"kubernetes.io/projected/faf178d8-2a71-4130-a786-4f31c15a5f33-kube-api-access-wf4j2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:56:48.218939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.218905 2519 generic.go:356] "Generic (PLEG): container finished" podID="faf178d8-2a71-4130-a786-4f31c15a5f33" containerID="23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c" exitCode=0 Sep 11 13:56:48.219383 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.218949 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-q26gg" Sep 11 13:56:48.219383 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.218992 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q26gg" event={"ID":"faf178d8-2a71-4130-a786-4f31c15a5f33","Type":"ContainerDied","Data":"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c"} Sep 11 13:56:48.219383 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.219041 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-q26gg" event={"ID":"faf178d8-2a71-4130-a786-4f31c15a5f33","Type":"ContainerDied","Data":"e228e8e3edb8d2326e4fcc3f1bb02623e6566077b5e8380308aabe80ada11ae3"} Sep 11 13:56:48.219383 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.219062 2519 scope.go:117] "RemoveContainer" containerID="23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c" Sep 11 13:56:48.228275 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.228256 2519 scope.go:117] "RemoveContainer" containerID="23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c" Sep 11 13:56:48.228514 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:48.228492 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c\": container with ID starting with 23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c not found: ID does not exist" containerID="23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c" Sep 11 13:56:48.228600 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.228526 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c"} err="failed to get container status \"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c\": rpc error: code = NotFound desc = could not find container \"23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c\": container with ID starting with 23dfbbfd9e631f3da03eaefec4e1453f81b439e872c3eeffa17d6d4efbfb9b7c not found: ID does not exist" Sep 11 13:56:48.239339 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.239312 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:56:48.243191 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.243170 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-q26gg"] Sep 11 13:56:48.690589 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.690548 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cf52a7-5376-4bfd-bf75-c93608e58825" path="/var/lib/kubelet/pods/32cf52a7-5376-4bfd-bf75-c93608e58825/volumes" Sep 11 13:56:48.690842 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.690830 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d3d929-c797-48dc-9c61-8d4db149d591" path="/var/lib/kubelet/pods/63d3d929-c797-48dc-9c61-8d4db149d591/volumes" Sep 11 13:56:48.691028 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:48.691017 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf178d8-2a71-4130-a786-4f31c15a5f33" path="/var/lib/kubelet/pods/faf178d8-2a71-4130-a786-4f31c15a5f33/volumes" Sep 11 13:56:51.924191 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.924161 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:51.950330 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.950291 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:56:51.950724 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.950711 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf178d8-2a71-4130-a786-4f31c15a5f33" containerName="authorino" Sep 11 13:56:51.950767 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.950726 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf178d8-2a71-4130-a786-4f31c15a5f33" containerName="authorino" Sep 11 13:56:51.950806 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.950795 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf178d8-2a71-4130-a786-4f31c15a5f33" containerName="authorino" Sep 11 13:56:51.955130 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.955110 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:51.959855 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.959832 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 13:56:51.960058 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.960039 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 13:56:51.960135 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.960060 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hrdh6\"" Sep 11 13:56:51.960672 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.960653 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 13:56:51.968299 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:51.968275 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:56:52.023975 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.023947 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:52.041564 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.041536 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.041671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.041593 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.041671 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.041647 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6sr\" (UniqueName: \"kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.125796 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.125756 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:52.132348 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.132317 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 13:56:52.135858 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.135835 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.142305 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.142122 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.142305 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.142174 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.142305 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.142198 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6sr\" (UniqueName: \"kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.142686 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:52.142532 2519 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 13:56:52.142686 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:52.142647 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert podName:70ec644b-4a94-4ec4-94ab-8519dd5c3091 nodeName:}" failed. No retries permitted until 2026-09-11 13:56:52.642626071 +0000 UTC m=+480.396940947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert") pod "maas-controller-7c9f64f957-ktt7v" (UID: "70ec644b-4a94-4ec4-94ab-8519dd5c3091") : secret "maas-controller-webhook-cert" not found Sep 11 13:56:52.145328 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.145308 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.146304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.146283 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 13:56:52.156511 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.156484 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6sr\" (UniqueName: \"kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.199184 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.199113 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:52.243123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.243097 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmszp\" (UniqueName: \"kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.243123 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.243126 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.243327 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.243209 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.344493 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.344462 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.344695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.344542 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmszp\" (UniqueName: \"kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.344695 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.344567 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.344773 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:52.344720 2519 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 13:56:52.344808 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:56:52.344780 2519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert podName:97c2da0a-653c-4f0c-9b4d-41c193911017 nodeName:}" failed. No retries permitted until 2026-09-11 13:56:52.844762716 +0000 UTC m=+480.599077586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert") pod "maas-controller-5cd946d98f-p59w5" (UID: "97c2da0a-653c-4f0c-9b4d-41c193911017") : secret "maas-controller-webhook-cert" not found Sep 11 13:56:52.346902 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.346875 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.352686 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.352662 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmszp\" (UniqueName: \"kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.551415 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.551329 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:52.647969 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.647934 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.650390 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.650359 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") pod \"maas-controller-7c9f64f957-ktt7v\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:52.850613 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.850542 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.852945 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.852918 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") pod \"maas-controller-5cd946d98f-p59w5\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:52.868013 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.867994 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hrdh6\"" Sep 11 13:56:52.875642 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:52.875624 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:53.047439 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:53.047401 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:53.150249 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:53.150223 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:56:53.152367 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:56:53.152338 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ec644b_4a94_4ec4_94ab_8519dd5c3091.slice/crio-045f4579c83c291cf7dc588bb955ada6ed6d198e5f60b4af721f760c07b0f494 WatchSource:0}: Error finding container 045f4579c83c291cf7dc588bb955ada6ed6d198e5f60b4af721f760c07b0f494: Status 404 returned error can't find the container with id 045f4579c83c291cf7dc588bb955ada6ed6d198e5f60b4af721f760c07b0f494 Sep 11 13:56:53.244109 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:53.244074 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" event={"ID":"70ec644b-4a94-4ec4-94ab-8519dd5c3091","Type":"ContainerStarted","Data":"045f4579c83c291cf7dc588bb955ada6ed6d198e5f60b4af721f760c07b0f494"} Sep 11 13:56:53.320981 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:53.320957 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 13:56:53.323241 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:56:53.323212 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c2da0a_653c_4f0c_9b4d_41c193911017.slice/crio-bcd99c280f9766f761bc8cffe405501e58accd7e65e2ca3f0f431bd6c485c89c WatchSource:0}: Error finding container bcd99c280f9766f761bc8cffe405501e58accd7e65e2ca3f0f431bd6c485c89c: Status 404 returned error can't find the container with id bcd99c280f9766f761bc8cffe405501e58accd7e65e2ca3f0f431bd6c485c89c Sep 11 13:56:54.257425 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:54.257387 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-p59w5" event={"ID":"97c2da0a-653c-4f0c-9b4d-41c193911017","Type":"ContainerStarted","Data":"bcd99c280f9766f761bc8cffe405501e58accd7e65e2ca3f0f431bd6c485c89c"} Sep 11 13:56:55.757184 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:55.757159 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 13:56:55.935802 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:55.935769 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:56.036645 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.036511 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:56.268822 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.268784 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" event={"ID":"70ec644b-4a94-4ec4-94ab-8519dd5c3091","Type":"ContainerStarted","Data":"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208"} Sep 11 13:56:56.269012 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.268910 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:56:56.270503 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.270480 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-p59w5" event={"ID":"97c2da0a-653c-4f0c-9b4d-41c193911017","Type":"ContainerStarted","Data":"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61"} Sep 11 13:56:56.270627 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.270522 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:56:56.289262 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.289172 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" podStartSLOduration=2.688227413 podStartE2EDuration="5.289158328s" totalImagesPullingTime="2.600930915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:56.286804678 +0000 UTC m=+484.041119573" watchObservedRunningTime="2026-09-11 13:56:56.289158328 +0000 UTC m=+484.043473216" Sep 11 13:56:56.308891 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:56:56.308835 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd946d98f-p59w5" podStartSLOduration=1.8711561620000001 podStartE2EDuration="4.308818916s" totalImagesPullingTime="2.437662754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:56.30779738 +0000 UTC m=+484.062112274" watchObservedRunningTime="2026-09-11 13:56:56.308818916 +0000 UTC m=+484.063133806" Sep 11 13:57:07.277929 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.277898 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:57:07.278469 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.277965 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 13:57:07.343585 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.343541 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:57:07.343801 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.343774 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" podUID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" containerName="manager" containerID="cri-o://b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208" gracePeriod=30 Sep 11 13:57:07.578616 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.578571 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:57:07.595089 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.595061 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k6sr\" (UniqueName: \"kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr\") pod \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " Sep 11 13:57:07.595220 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.595146 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") pod \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " Sep 11 13:57:07.595220 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.595185 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs\") pod \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\" (UID: \"70ec644b-4a94-4ec4-94ab-8519dd5c3091\") " Sep 11 13:57:07.597991 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.597955 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr" (OuterVolumeSpecName: "kube-api-access-4k6sr") pod "70ec644b-4a94-4ec4-94ab-8519dd5c3091" (UID: "70ec644b-4a94-4ec4-94ab-8519dd5c3091"). InnerVolumeSpecName "kube-api-access-4k6sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:57:07.597991 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.597968 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "70ec644b-4a94-4ec4-94ab-8519dd5c3091" (UID: "70ec644b-4a94-4ec4-94ab-8519dd5c3091"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:57:07.598135 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.598070 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70ec644b-4a94-4ec4-94ab-8519dd5c3091" (UID: "70ec644b-4a94-4ec4-94ab-8519dd5c3091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:57:07.696170 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.696133 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6sr\" (UniqueName: \"kubernetes.io/projected/70ec644b-4a94-4ec4-94ab-8519dd5c3091-kube-api-access-4k6sr\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:57:07.696170 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.696161 2519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-webhook-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:57:07.696170 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:07.696171 2519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ec644b-4a94-4ec4-94ab-8519dd5c3091-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 13:57:08.335410 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.335372 2519 generic.go:356] "Generic (PLEG): container finished" podID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" containerID="b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208" exitCode=0 Sep 11 13:57:08.335841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.335419 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" Sep 11 13:57:08.335841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.335458 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" event={"ID":"70ec644b-4a94-4ec4-94ab-8519dd5c3091","Type":"ContainerDied","Data":"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208"} Sep 11 13:57:08.335841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.335506 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c9f64f957-ktt7v" event={"ID":"70ec644b-4a94-4ec4-94ab-8519dd5c3091","Type":"ContainerDied","Data":"045f4579c83c291cf7dc588bb955ada6ed6d198e5f60b4af721f760c07b0f494"} Sep 11 13:57:08.335841 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.335530 2519 scope.go:117] "RemoveContainer" containerID="b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208" Sep 11 13:57:08.345043 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.344890 2519 scope.go:117] "RemoveContainer" containerID="b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208" Sep 11 13:57:08.345172 ip-10-0-139-201 kubenswrapper[2519]: E0911 13:57:08.345151 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208\": container with ID starting with b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208 not found: ID does not exist" containerID="b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208" Sep 11 13:57:08.345212 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.345181 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208"} err="failed to get container status \"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208\": rpc error: code = NotFound desc = could not find container \"b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208\": container with ID starting with b6bad175f80912782f1e8d11ffe0b0653aae8b56b6b7e785120e5443b27ae208 not found: ID does not exist" Sep 11 13:57:08.356883 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.356859 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:57:08.360955 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.360935 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c9f64f957-ktt7v"] Sep 11 13:57:08.690699 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:08.690622 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" path="/var/lib/kubelet/pods/70ec644b-4a94-4ec4-94ab-8519dd5c3091/volumes" Sep 11 13:57:12.532820 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:12.532780 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:12.634095 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:12.634049 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:27.294690 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:27.294650 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:27.350093 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:27.350061 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:27.448530 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:27.448498 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:28.564565 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:28.564473 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:28.659984 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:28.659947 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.147113 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.147070 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.249490 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.249451 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.347814 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.347780 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.484494 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.484461 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.724043 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.723967 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.784627 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.784592 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv"] Sep 11 13:57:30.784983 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.784970 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" containerName="manager" Sep 11 13:57:30.785029 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.784985 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" containerName="manager" Sep 11 13:57:30.785078 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.785069 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ec644b-4a94-4ec4-94ab-8519dd5c3091" containerName="manager" Sep 11 13:57:30.790217 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.790192 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:30.791339 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.791322 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:30.793295 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.793272 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 13:57:30.793413 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.793306 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jrw7w\"" Sep 11 13:57:30.801085 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.801050 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv"] Sep 11 13:57:30.819121 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.819097 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2s2j"] Sep 11 13:57:30.822215 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.822197 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:30.832906 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.832884 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2s2j"] Sep 11 13:57:30.916112 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.916067 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkqz\" (UniqueName: \"kubernetes.io/projected/b42aca23-72e9-48f7-b863-2e10b4658e2f-kube-api-access-6rkqz\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:30.916306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.916157 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9tbk\" (UniqueName: \"kubernetes.io/projected/a526c097-255f-4589-b80e-cd62afaa9128-kube-api-access-v9tbk\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:30.916306 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.916282 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a526c097-255f-4589-b80e-cd62afaa9128-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:30.916413 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:30.916318 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42aca23-72e9-48f7-b863-2e10b4658e2f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.017609 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.017514 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkqz\" (UniqueName: \"kubernetes.io/projected/b42aca23-72e9-48f7-b863-2e10b4658e2f-kube-api-access-6rkqz\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.017609 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.017556 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9tbk\" (UniqueName: \"kubernetes.io/projected/a526c097-255f-4589-b80e-cd62afaa9128-kube-api-access-v9tbk\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:31.017858 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.017635 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a526c097-255f-4589-b80e-cd62afaa9128-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:31.017858 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.017657 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42aca23-72e9-48f7-b863-2e10b4658e2f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.018462 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.018439 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42aca23-72e9-48f7-b863-2e10b4658e2f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.018511 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.018464 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a526c097-255f-4589-b80e-cd62afaa9128-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:31.031956 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.031913 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9tbk\" (UniqueName: \"kubernetes.io/projected/a526c097-255f-4589-b80e-cd62afaa9128-kube-api-access-v9tbk\") pod \"payload-processing-5b54b7fbb7-z2s2j\" (UID: \"a526c097-255f-4589-b80e-cd62afaa9128\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:31.032325 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.032306 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkqz\" (UniqueName: \"kubernetes.io/projected/b42aca23-72e9-48f7-b863-2e10b4658e2f-kube-api-access-6rkqz\") pod \"payload-pre-processing-59fbc47bf8-rbcpv\" (UID: \"b42aca23-72e9-48f7-b863-2e10b4658e2f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.102184 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.102146 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:31.132257 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.132221 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:31.379362 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.379335 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv"] Sep 11 13:57:31.381506 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:57:31.381478 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42aca23_72e9_48f7_b863_2e10b4658e2f.slice/crio-a6505ac3b9f8e1887599553def09698cf9b017f5993c3c0e460821fb1912832f WatchSource:0}: Error finding container a6505ac3b9f8e1887599553def09698cf9b017f5993c3c0e460821fb1912832f: Status 404 returned error can't find the container with id a6505ac3b9f8e1887599553def09698cf9b017f5993c3c0e460821fb1912832f Sep 11 13:57:31.419709 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.419686 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-z2s2j"] Sep 11 13:57:31.421339 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:57:31.421314 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda526c097_255f_4589_b80e_cd62afaa9128.slice/crio-30d00aef109cf0d951334f6920a2f6876c2aa5c4e03f5b7bd4bc44f21c59f4bc WatchSource:0}: Error finding container 30d00aef109cf0d951334f6920a2f6876c2aa5c4e03f5b7bd4bc44f21c59f4bc: Status 404 returned error can't find the container with id 30d00aef109cf0d951334f6920a2f6876c2aa5c4e03f5b7bd4bc44f21c59f4bc Sep 11 13:57:31.446286 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.446254 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" event={"ID":"b42aca23-72e9-48f7-b863-2e10b4658e2f","Type":"ContainerStarted","Data":"a6505ac3b9f8e1887599553def09698cf9b017f5993c3c0e460821fb1912832f"} Sep 11 13:57:31.447449 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:31.447429 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" event={"ID":"a526c097-255f-4589-b80e-cd62afaa9128","Type":"ContainerStarted","Data":"30d00aef109cf0d951334f6920a2f6876c2aa5c4e03f5b7bd4bc44f21c59f4bc"} Sep 11 13:57:33.461847 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.461746 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" event={"ID":"b42aca23-72e9-48f7-b863-2e10b4658e2f","Type":"ContainerStarted","Data":"1970cb71617cde7378b131245a103110b3dbf942c5bd01c59eb41093904eabdd"} Sep 11 13:57:33.461847 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.461826 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:33.463605 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.463567 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" event={"ID":"a526c097-255f-4589-b80e-cd62afaa9128","Type":"ContainerStarted","Data":"f36f7f088e8d7ffb721e9a5040e07482677798aedbf5d9dbfa7516a389ec1dc2"} Sep 11 13:57:33.463722 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.463655 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:33.479830 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.479770 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" podStartSLOduration=1.648217717 podStartE2EDuration="3.479750821s" totalImagesPullingTime="1.831533104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:57:33.478461898 +0000 UTC m=+521.232776787" watchObservedRunningTime="2026-09-11 13:57:33.479750821 +0000 UTC m=+521.234065717" Sep 11 13:57:33.496343 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:33.496288 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" podStartSLOduration=1.6971437040000001 podStartE2EDuration="3.496267876s" totalImagesPullingTime="1.799124172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:57:33.493757935 +0000 UTC m=+521.248072828" watchObservedRunningTime="2026-09-11 13:57:33.496267876 +0000 UTC m=+521.250582769" Sep 11 13:57:40.364125 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:40.364086 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:40.458835 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:40.458801 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:44.471115 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:44.471084 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-z2s2j" Sep 11 13:57:44.471514 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:44.471153 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rbcpv" Sep 11 13:57:53.197114 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:57:53.197037 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:08.753488 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.753453 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts"] Sep 11 13:58:08.759730 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.759710 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.762088 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.762064 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p2lbh\"" Sep 11 13:58:08.762194 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.762064 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 11 13:58:08.762194 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.762067 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 13:58:08.762469 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.762455 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 13:58:08.766178 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.766154 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts"] Sep 11 13:58:08.796677 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.796650 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:08.875012 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.874978 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.875012 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.875015 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.875229 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.875052 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96snd\" (UniqueName: \"kubernetes.io/projected/47f15cbc-b7a8-4b61-b21b-d6201974bcad-kube-api-access-96snd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.875229 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.875137 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.875229 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.875212 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.875350 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.875239 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976736 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976701 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976772 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976802 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976833 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976851 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.976939 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.976895 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96snd\" (UniqueName: \"kubernetes.io/projected/47f15cbc-b7a8-4b61-b21b-d6201974bcad-kube-api-access-96snd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.977225 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.977191 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.977298 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.977218 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.977298 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.977290 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.979193 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.979170 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f15cbc-b7a8-4b61-b21b-d6201974bcad-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.979392 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.979377 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f15cbc-b7a8-4b61-b21b-d6201974bcad-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:08.984513 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:08.984491 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96snd\" (UniqueName: \"kubernetes.io/projected/47f15cbc-b7a8-4b61-b21b-d6201974bcad-kube-api-access-96snd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jh8ts\" (UID: \"47f15cbc-b7a8-4b61-b21b-d6201974bcad\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:09.071824 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:09.071736 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:09.342144 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:09.342120 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts"] Sep 11 13:58:09.344081 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:58:09.344052 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f15cbc_b7a8_4b61_b21b_d6201974bcad.slice/crio-93da2834a98bf934f9e75178e3375e1987326624bad5189a5bd6e29eeba195e3 WatchSource:0}: Error finding container 93da2834a98bf934f9e75178e3375e1987326624bad5189a5bd6e29eeba195e3: Status 404 returned error can't find the container with id 93da2834a98bf934f9e75178e3375e1987326624bad5189a5bd6e29eeba195e3 Sep 11 13:58:09.647613 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:09.647508 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" event={"ID":"47f15cbc-b7a8-4b61-b21b-d6201974bcad","Type":"ContainerStarted","Data":"93da2834a98bf934f9e75178e3375e1987326624bad5189a5bd6e29eeba195e3"} Sep 11 13:58:10.654004 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:10.653971 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" event={"ID":"47f15cbc-b7a8-4b61-b21b-d6201974bcad","Type":"ContainerStarted","Data":"f4b6e69a64af57892373158780993afb069c5e2777cebbf25bdf54d2a912bb59"} Sep 11 13:58:10.672081 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:10.672031 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" podStartSLOduration=1.520778832 podStartE2EDuration="2.672016673s" totalImagesPullingTime="1.151237841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:10.670347634 +0000 UTC m=+558.424662525" watchObservedRunningTime="2026-09-11 13:58:10.672016673 +0000 UTC m=+558.426331565" Sep 11 13:58:19.072114 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:19.072073 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:19.072674 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:19.072151 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:19.084778 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:19.084751 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:19.708931 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:19.708901 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jh8ts" Sep 11 13:58:49.896891 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:49.896854 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:52.669482 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:52.669452 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:58:52.669948 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:58:52.669525 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 13:59:04.344556 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:04.344521 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.365213 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.365179 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c"] Sep 11 13:59:06.368520 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.368498 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.370677 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.370658 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 13:59:06.377662 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.377640 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c"] Sep 11 13:59:06.401199 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401167 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.401396 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401242 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.401396 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401277 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5rn\" (UniqueName: \"kubernetes.io/projected/905be396-da4c-4ae9-9e3c-f52ea0b5e809-kube-api-access-tn5rn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.401396 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401351 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.401543 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401399 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.401543 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.401425 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.469737 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.469704 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.502046 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502011 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502054 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5rn\" (UniqueName: \"kubernetes.io/projected/905be396-da4c-4ae9-9e3c-f52ea0b5e809-kube-api-access-tn5rn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502104 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502139 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502235 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502155 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502463 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502304 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502603 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502551 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502681 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502659 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.502728 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.502659 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.504452 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.504432 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/905be396-da4c-4ae9-9e3c-f52ea0b5e809-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.504710 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.504694 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/905be396-da4c-4ae9-9e3c-f52ea0b5e809-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.510095 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.510065 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5rn\" (UniqueName: \"kubernetes.io/projected/905be396-da4c-4ae9-9e3c-f52ea0b5e809-kube-api-access-tn5rn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zkx9c\" (UID: \"905be396-da4c-4ae9-9e3c-f52ea0b5e809\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.680304 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.680205 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:06.951107 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:06.951082 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c"] Sep 11 13:59:06.953261 ip-10-0-139-201 kubenswrapper[2519]: W0911 13:59:06.953230 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905be396_da4c_4ae9_9e3c_f52ea0b5e809.slice/crio-c4c5781740989601526a870a55ac3732fed34f2ea3c02fe357a8285eb92e838c WatchSource:0}: Error finding container c4c5781740989601526a870a55ac3732fed34f2ea3c02fe357a8285eb92e838c: Status 404 returned error can't find the container with id c4c5781740989601526a870a55ac3732fed34f2ea3c02fe357a8285eb92e838c Sep 11 13:59:07.932994 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:07.932958 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" event={"ID":"905be396-da4c-4ae9-9e3c-f52ea0b5e809","Type":"ContainerStarted","Data":"84b7a54902f1eacc969640d122a584c5ea43bd763f1475abe01f06f72e678275"} Sep 11 13:59:07.933412 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:07.933012 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" event={"ID":"905be396-da4c-4ae9-9e3c-f52ea0b5e809","Type":"ContainerStarted","Data":"c4c5781740989601526a870a55ac3732fed34f2ea3c02fe357a8285eb92e838c"} Sep 11 13:59:07.950061 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:07.950010 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" podStartSLOduration=1.7646883309999999 podStartE2EDuration="1.949991139s" totalImagesPullingTime="185.302808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:07.948783194 +0000 UTC m=+615.703098085" watchObservedRunningTime="2026-09-11 13:59:07.949991139 +0000 UTC m=+615.704306048" Sep 11 13:59:11.346757 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:11.346727 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:11.404238 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:11.404209 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:15.465331 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:15.465253 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:16.681065 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:16.681020 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:16.681540 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:16.681114 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:16.693988 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:16.693958 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:16.990270 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:16.990194 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zkx9c" Sep 11 13:59:34.699938 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:34.699908 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:45.494245 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:45.494180 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:57.363897 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:57.363866 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:57.460803 ip-10-0-139-201 kubenswrapper[2519]: I0911 13:59:57.460771 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:00.146207 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.146170 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq"] Sep 11 14:00:00.149532 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.149514 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" Sep 11 14:00:00.151567 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.151533 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:00:00.156080 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.156053 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq"] Sep 11 14:00:00.160520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.160492 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" Sep 11 14:00:00.437989 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:00.437961 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq"] Sep 11 14:00:00.440888 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:00:00.440860 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf9a634_ad29_4043_b115_ef68f521fa92.slice/crio-5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45 WatchSource:0}: Error finding container 5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45: Status 404 returned error can't find the container with id 5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45 Sep 11 14:00:01.198777 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:01.198743 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" event={"ID":"1cf9a634-ad29-4043-b115-ef68f521fa92","Type":"ContainerStarted","Data":"5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45"} Sep 11 14:00:15.271009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:15.270975 2519 generic.go:356] "Generic (PLEG): container finished" podID="1cf9a634-ad29-4043-b115-ef68f521fa92" containerID="195fc435a6b0daa342b1748eade93993581c5e7ecfd5e05d28f686f277cb922f" exitCode=0 Sep 11 14:00:15.271412 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:15.271062 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" event={"ID":"1cf9a634-ad29-4043-b115-ef68f521fa92","Type":"ContainerDied","Data":"195fc435a6b0daa342b1748eade93993581c5e7ecfd5e05d28f686f277cb922f"} Sep 11 14:00:16.411111 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:16.411086 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" Sep 11 14:00:17.282694 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:17.282668 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" Sep 11 14:00:17.282694 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:17.282680 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818920-r5kxq" event={"ID":"1cf9a634-ad29-4043-b115-ef68f521fa92","Type":"ContainerDied","Data":"5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45"} Sep 11 14:00:17.282913 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:00:17.282732 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5704c0ee78ccde6721920e6e89438196ce58dcfb2c2fbc49b8d4d98881713b45" Sep 11 14:01:06.598727 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.598651 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56fb644598-rb4x5"] Sep 11 14:01:06.599239 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.599084 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf9a634-ad29-4043-b115-ef68f521fa92" containerName="cleanup" Sep 11 14:01:06.599239 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.599095 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf9a634-ad29-4043-b115-ef68f521fa92" containerName="cleanup" Sep 11 14:01:06.599239 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.599165 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf9a634-ad29-4043-b115-ef68f521fa92" containerName="cleanup" Sep 11 14:01:06.605814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.605795 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.615355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.615328 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb644598-rb4x5"] Sep 11 14:01:06.636843 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.636819 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f91fa401-a2cd-4356-adb7-b2eb01cd962c-tls-cert\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.636961 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.636948 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmnl\" (UniqueName: \"kubernetes.io/projected/f91fa401-a2cd-4356-adb7-b2eb01cd962c-kube-api-access-dtmnl\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.738388 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.738345 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmnl\" (UniqueName: \"kubernetes.io/projected/f91fa401-a2cd-4356-adb7-b2eb01cd962c-kube-api-access-dtmnl\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.738605 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.738428 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f91fa401-a2cd-4356-adb7-b2eb01cd962c-tls-cert\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.740878 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.740852 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f91fa401-a2cd-4356-adb7-b2eb01cd962c-tls-cert\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.745777 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.745753 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmnl\" (UniqueName: \"kubernetes.io/projected/f91fa401-a2cd-4356-adb7-b2eb01cd962c-kube-api-access-dtmnl\") pod \"authorino-56fb644598-rb4x5\" (UID: \"f91fa401-a2cd-4356-adb7-b2eb01cd962c\") " pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:06.915890 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:06.915805 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb644598-rb4x5" Sep 11 14:01:07.184930 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:07.184906 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb644598-rb4x5"] Sep 11 14:01:07.187068 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:07.187044 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf91fa401_a2cd_4356_adb7_b2eb01cd962c.slice/crio-cf0392569569c0cf9a63d219bdfab76088331f90e18def38838422d1768296b7 WatchSource:0}: Error finding container cf0392569569c0cf9a63d219bdfab76088331f90e18def38838422d1768296b7: Status 404 returned error can't find the container with id cf0392569569c0cf9a63d219bdfab76088331f90e18def38838422d1768296b7 Sep 11 14:01:07.188662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:07.188643 2519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:01:07.531377 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:07.531257 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb644598-rb4x5" event={"ID":"f91fa401-a2cd-4356-adb7-b2eb01cd962c","Type":"ContainerStarted","Data":"cf0392569569c0cf9a63d219bdfab76088331f90e18def38838422d1768296b7"} Sep 11 14:01:09.543139 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.543099 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb644598-rb4x5" event={"ID":"f91fa401-a2cd-4356-adb7-b2eb01cd962c","Type":"ContainerStarted","Data":"3fd047ac09286dac838f15addc2b92fd9c0786dba899eb9a68c2fb0bdcd094f1"} Sep 11 14:01:09.562071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.562018 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56fb644598-rb4x5" podStartSLOduration=1.680609201 podStartE2EDuration="3.562001513s" totalImagesPullingTime="1.881392312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:09.560566949 +0000 UTC m=+737.314881841" watchObservedRunningTime="2026-09-11 14:01:09.562001513 +0000 UTC m=+737.316316404" Sep 11 14:01:09.590425 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.590380 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 14:01:09.590663 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.590633 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" podUID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" containerName="authorino" containerID="cri-o://6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b" gracePeriod=30 Sep 11 14:01:09.848337 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.848308 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 14:01:09.974118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.974083 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert\") pod \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " Sep 11 14:01:09.974300 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.974167 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77n86\" (UniqueName: \"kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86\") pod \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\" (UID: \"257cd4ee-2b2d-4191-a17d-1fae6d689d62\") " Sep 11 14:01:09.976230 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.976198 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86" (OuterVolumeSpecName: "kube-api-access-77n86") pod "257cd4ee-2b2d-4191-a17d-1fae6d689d62" (UID: "257cd4ee-2b2d-4191-a17d-1fae6d689d62"). InnerVolumeSpecName "kube-api-access-77n86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:01:09.985570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:09.985534 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "257cd4ee-2b2d-4191-a17d-1fae6d689d62" (UID: "257cd4ee-2b2d-4191-a17d-1fae6d689d62"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:01:10.075926 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.075883 2519 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/257cd4ee-2b2d-4191-a17d-1fae6d689d62-tls-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:01:10.076067 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.075930 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77n86\" (UniqueName: \"kubernetes.io/projected/257cd4ee-2b2d-4191-a17d-1fae6d689d62-kube-api-access-77n86\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:01:10.136142 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.136107 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:10.243264 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.243171 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:10.551320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.551236 2519 generic.go:356] "Generic (PLEG): container finished" podID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" containerID="6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b" exitCode=0 Sep 11 14:01:10.551320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.551307 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" Sep 11 14:01:10.551796 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.551329 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" event={"ID":"257cd4ee-2b2d-4191-a17d-1fae6d689d62","Type":"ContainerDied","Data":"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b"} Sep 11 14:01:10.551796 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.551376 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498fdbcf5-nzxgh" event={"ID":"257cd4ee-2b2d-4191-a17d-1fae6d689d62","Type":"ContainerDied","Data":"5b9fa07a037c63fc18cfbbaaa500a2f3b22c8f479da01cc2eeed3aaf5ca0024d"} Sep 11 14:01:10.551796 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.551394 2519 scope.go:117] "RemoveContainer" containerID="6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b" Sep 11 14:01:10.560210 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.560190 2519 scope.go:117] "RemoveContainer" containerID="6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b" Sep 11 14:01:10.560480 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:10.560457 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b\": container with ID starting with 6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b not found: ID does not exist" containerID="6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b" Sep 11 14:01:10.560562 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.560492 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b"} err="failed to get container status \"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b\": rpc error: code = NotFound desc = could not find container \"6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b\": container with ID starting with 6ff76c16d9d2450b0f4099f06b450513d369357027eb63833b47f2bd7095034b not found: ID does not exist" Sep 11 14:01:10.574846 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.574821 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 14:01:10.579137 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.579115 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7498fdbcf5-nzxgh"] Sep 11 14:01:10.691343 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:10.691312 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" path="/var/lib/kubelet/pods/257cd4ee-2b2d-4191-a17d-1fae6d689d62/volumes" Sep 11 14:01:11.355393 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:11.355350 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:11.453368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:11.453333 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:29.447436 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:29.447405 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:29.560570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:29.560528 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:29.831756 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:29.831678 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:30.324771 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:30.324731 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:30.934718 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:30.934676 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:31.029275 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.029245 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:31.379657 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.379624 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:31.571667 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.571626 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-b4c695"] Sep 11 14:01:31.572092 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.572074 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" containerName="authorino" Sep 11 14:01:31.572184 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.572095 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" containerName="authorino" Sep 11 14:01:31.572247 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.572230 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="257cd4ee-2b2d-4191-a17d-1fae6d689d62" containerName="authorino" Sep 11 14:01:31.575494 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.575474 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:31.578164 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.578142 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qcb7\"" Sep 11 14:01:31.579430 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.579407 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-b4c695"] Sep 11 14:01:31.686483 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.686386 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcx2\" (UniqueName: \"kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2\") pod \"test-curl-7552-b4c695\" (UID: \"345111ca-551e-4e05-b43b-07e12f3af6cf\") " pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:31.787345 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.787307 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcx2\" (UniqueName: \"kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2\") pod \"test-curl-7552-b4c695\" (UID: \"345111ca-551e-4e05-b43b-07e12f3af6cf\") " pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:31.795119 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.795094 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcx2\" (UniqueName: \"kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2\") pod \"test-curl-7552-b4c695\" (UID: \"345111ca-551e-4e05-b43b-07e12f3af6cf\") " pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:31.886375 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:31.886343 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:32.162072 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:32.162044 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-b4c695"] Sep 11 14:01:32.164534 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:32.164506 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod345111ca_551e_4e05_b43b_07e12f3af6cf.slice/crio-37f125d2ae7ec371a734ab41a4005cab055444259d24a1e3511e5e52cea78850 WatchSource:0}: Error finding container 37f125d2ae7ec371a734ab41a4005cab055444259d24a1e3511e5e52cea78850: Status 404 returned error can't find the container with id 37f125d2ae7ec371a734ab41a4005cab055444259d24a1e3511e5e52cea78850 Sep 11 14:01:32.552277 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:32.552245 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:32.661873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:32.661840 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-b4c695" event={"ID":"345111ca-551e-4e05-b43b-07e12f3af6cf","Type":"ContainerStarted","Data":"37f125d2ae7ec371a734ab41a4005cab055444259d24a1e3511e5e52cea78850"} Sep 11 14:01:33.300589 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:33.300539 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:33.380002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:33.379967 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:34.527267 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:34.527230 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:34.642043 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:34.642004 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:34.675051 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:34.674960 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-b4c695" event={"ID":"345111ca-551e-4e05-b43b-07e12f3af6cf","Type":"ContainerStarted","Data":"1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9"} Sep 11 14:01:34.691470 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:34.691415 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7552-b4c695" podStartSLOduration=1.459325637 podStartE2EDuration="3.69139862s" totalImagesPullingTime="2.232072983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:34.689614452 +0000 UTC m=+762.443929344" watchObservedRunningTime="2026-09-11 14:01:34.69139862 +0000 UTC m=+762.445713511" Sep 11 14:01:34.839773 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:34.839738 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:35.064033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.063957 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:35.166336 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.166304 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:35.184672 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.184638 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:01:35.188466 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.188445 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.190842 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.190815 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9c9fc1\"" Sep 11 14:01:35.197558 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.197535 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:01:35.223103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.223061 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.223276 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.223200 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswk4\" (UniqueName: \"kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.311095 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.311062 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-b4c695"] Sep 11 14:01:35.324785 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.324700 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.324785 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.324742 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-b4c695"] Sep 11 14:01:35.325011 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.324831 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswk4\" (UniqueName: \"kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.325474 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.325451 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.332266 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.332240 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:01:35.332690 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.332676 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" containerName="test-curl-7552-b4c695" Sep 11 14:01:35.332731 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.332691 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" containerName="test-curl-7552-b4c695" Sep 11 14:01:35.332776 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.332764 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" containerName="test-curl-7552-b4c695" Sep 11 14:01:35.335938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.335921 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.338132 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.338109 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9c9fc1-dockercfg-mlbll\"" Sep 11 14:01:35.339386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.339366 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswk4\" (UniqueName: \"kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4\") pod \"payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.351485 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.351460 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:01:35.425778 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.425740 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn4g\" (UniqueName: \"kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.425778 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.425780 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.438806 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.438767 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-7754a4"] Sep 11 14:01:35.442327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.442303 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:35.452143 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.452119 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-7754a4"] Sep 11 14:01:35.500701 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.500665 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:35.526623 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.526597 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dlqf\" (UniqueName: \"kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf\") pod \"test-curl-7552-7754a4\" (UID: \"4850dd70-bed8-4346-a23b-ab24bea28da9\") " pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:35.526722 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.526652 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn4g\" (UniqueName: \"kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.526722 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.526708 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.527585 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.527549 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.535231 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.535197 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn4g\" (UniqueName: \"kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g\") pod \"payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.627678 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.627644 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlqf\" (UniqueName: \"kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf\") pod \"test-curl-7552-7754a4\" (UID: \"4850dd70-bed8-4346-a23b-ab24bea28da9\") " pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:35.638110 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.638080 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dlqf\" (UniqueName: \"kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf\") pod \"test-curl-7552-7754a4\" (UID: \"4850dd70-bed8-4346-a23b-ab24bea28da9\") " pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:35.658028 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.657987 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:35.680603 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:35.680554 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7552-b4c695" container="test-curl-7552-b4c695" Sep 11 14:01:35.752243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.752208 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:35.779097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.779062 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:01:35.781767 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:35.781737 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64ab1f4_c994_421f_8b6f_1fb204982703.slice/crio-bd7801d07b40a5c38e2733da5feee209309074c844c1b7431fc98d2283af7e1a WatchSource:0}: Error finding container bd7801d07b40a5c38e2733da5feee209309074c844c1b7431fc98d2283af7e1a: Status 404 returned error can't find the container with id bd7801d07b40a5c38e2733da5feee209309074c844c1b7431fc98d2283af7e1a Sep 11 14:01:35.931769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:35.931740 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:01:35.933488 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:35.933463 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975dd800_8d43_48ee_85ec_f2ce324d8086.slice/crio-80d8be9d2bc296167a0a459ddc71e99c7f51706d005e63947407a12e37136542 WatchSource:0}: Error finding container 80d8be9d2bc296167a0a459ddc71e99c7f51706d005e63947407a12e37136542: Status 404 returned error can't find the container with id 80d8be9d2bc296167a0a459ddc71e99c7f51706d005e63947407a12e37136542 Sep 11 14:01:36.028637 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.028612 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-7754a4"] Sep 11 14:01:36.030425 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:36.030395 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4850dd70_bed8_4346_a23b_ab24bea28da9.slice/crio-ad31fbabe4e6555b6a6525d38a1384307433b2a5a130954f2333094577f9744c WatchSource:0}: Error finding container ad31fbabe4e6555b6a6525d38a1384307433b2a5a130954f2333094577f9744c: Status 404 returned error can't find the container with id ad31fbabe4e6555b6a6525d38a1384307433b2a5a130954f2333094577f9744c Sep 11 14:01:36.692839 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.692799 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7552-b4c695" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" containerName="test-curl-7552-b4c695" containerID="cri-o://1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9" gracePeriod=2 Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.692966 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-7754a4" event={"ID":"4850dd70-bed8-4346-a23b-ab24bea28da9","Type":"ContainerStarted","Data":"d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16"} Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693000 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-7754a4" event={"ID":"4850dd70-bed8-4346-a23b-ab24bea28da9","Type":"ContainerStarted","Data":"ad31fbabe4e6555b6a6525d38a1384307433b2a5a130954f2333094577f9744c"} Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693021 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693049 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" event={"ID":"975dd800-8d43-48ee-85ec-f2ce324d8086","Type":"ContainerStarted","Data":"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6"} Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693068 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" event={"ID":"975dd800-8d43-48ee-85ec-f2ce324d8086","Type":"ContainerStarted","Data":"80d8be9d2bc296167a0a459ddc71e99c7f51706d005e63947407a12e37136542"} Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693088 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" event={"ID":"c64ab1f4-c994-421f-8b6f-1fb204982703","Type":"ContainerStarted","Data":"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556"} Sep 11 14:01:36.693327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.693109 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" event={"ID":"c64ab1f4-c994-421f-8b6f-1fb204982703","Type":"ContainerStarted","Data":"bd7801d07b40a5c38e2733da5feee209309074c844c1b7431fc98d2283af7e1a"} Sep 11 14:01:36.704049 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.703999 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7552-7754a4" podStartSLOduration=1.703980727 podStartE2EDuration="1.703980727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:36.702056025 +0000 UTC m=+764.456370915" watchObservedRunningTime="2026-09-11 14:01:36.703980727 +0000 UTC m=+764.458295618" Sep 11 14:01:36.716722 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.716679 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" podStartSLOduration=1.716665611 podStartE2EDuration="1.716665611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:36.715703508 +0000 UTC m=+764.470018410" watchObservedRunningTime="2026-09-11 14:01:36.716665611 +0000 UTC m=+764.470980507" Sep 11 14:01:36.731324 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:36.731279 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" podStartSLOduration=1.731265127 podStartE2EDuration="1.731265127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:36.730120751 +0000 UTC m=+764.484435641" watchObservedRunningTime="2026-09-11 14:01:36.731265127 +0000 UTC m=+764.485580018" Sep 11 14:01:37.364337 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.364253 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-7754a4"] Sep 11 14:01:37.368969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.368938 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-7754a4"] Sep 11 14:01:37.496485 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.496445 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:37.596972 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.596932 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:37.698961 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:37.698870 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7552-7754a4" container="test-curl-7552-7754a4" Sep 11 14:01:37.699321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.698878 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:37.813570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.813526 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:37.943397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:37.943358 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:38.037017 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.036932 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:38.097615 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.097585 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:38.253330 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.253298 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:38.705041 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.704986 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7552-7754a4" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" containerName="test-curl-7552-7754a4" containerID="cri-o://d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16" gracePeriod=2 Sep 11 14:01:38.732833 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.732790 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:38.864362 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.864339 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:38.961795 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.961706 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mcx2\" (UniqueName: \"kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2\") pod \"345111ca-551e-4e05-b43b-07e12f3af6cf\" (UID: \"345111ca-551e-4e05-b43b-07e12f3af6cf\") " Sep 11 14:01:38.963806 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:38.963783 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2" (OuterVolumeSpecName: "kube-api-access-4mcx2") pod "345111ca-551e-4e05-b43b-07e12f3af6cf" (UID: "345111ca-551e-4e05-b43b-07e12f3af6cf"). InnerVolumeSpecName "kube-api-access-4mcx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:01:39.062995 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.062963 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mcx2\" (UniqueName: \"kubernetes.io/projected/345111ca-551e-4e05-b43b-07e12f3af6cf-kube-api-access-4mcx2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:01:39.712661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.712628 2519 generic.go:356] "Generic (PLEG): container finished" podID="345111ca-551e-4e05-b43b-07e12f3af6cf" containerID="1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9" exitCode=137 Sep 11 14:01:39.713133 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.712671 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:39.713133 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.712714 2519 scope.go:117] "RemoveContainer" containerID="1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9" Sep 11 14:01:39.721508 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.721490 2519 scope.go:117] "RemoveContainer" containerID="1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9" Sep 11 14:01:39.721814 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:39.721795 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9\": container with ID starting with 1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9 not found: ID does not exist" containerID="1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9" Sep 11 14:01:39.721902 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:39.721822 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9"} err="failed to get container status \"1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9\": rpc error: code = NotFound desc = could not find container \"1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9\": container with ID starting with 1480f441c6895873b5acf264ee6e1aa36af212e873342f61f07a5e6d0fc8b6b9 not found: ID does not exist" Sep 11 14:01:39.723038 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:39.723016 2519 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7552-b4c695\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" pod="opendatahub/test-curl-7552-b4c695" Sep 11 14:01:40.693093 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:40.693061 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="345111ca-551e-4e05-b43b-07e12f3af6cf" path="/var/lib/kubelet/pods/345111ca-551e-4e05-b43b-07e12f3af6cf/volumes" Sep 11 14:01:40.849303 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:40.849269 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:40.897097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:40.897071 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:40.981071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:40.980987 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dlqf\" (UniqueName: \"kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf\") pod \"4850dd70-bed8-4346-a23b-ab24bea28da9\" (UID: \"4850dd70-bed8-4346-a23b-ab24bea28da9\") " Sep 11 14:01:40.983157 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:40.983125 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf" (OuterVolumeSpecName: "kube-api-access-4dlqf") pod "4850dd70-bed8-4346-a23b-ab24bea28da9" (UID: "4850dd70-bed8-4346-a23b-ab24bea28da9"). InnerVolumeSpecName "kube-api-access-4dlqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:01:41.008163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.008134 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:41.082827 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.082785 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dlqf\" (UniqueName: \"kubernetes.io/projected/4850dd70-bed8-4346-a23b-ab24bea28da9-kube-api-access-4dlqf\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:01:41.392287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.392251 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:41.492977 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.492940 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:41.727853 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.727730 2519 generic.go:356] "Generic (PLEG): container finished" podID="4850dd70-bed8-4346-a23b-ab24bea28da9" containerID="d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16" exitCode=137 Sep 11 14:01:41.727853 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.727804 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:41.727853 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.727819 2519 scope.go:117] "RemoveContainer" containerID="d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16" Sep 11 14:01:41.737727 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.737704 2519 scope.go:117] "RemoveContainer" containerID="d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16" Sep 11 14:01:41.738010 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:41.737991 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16\": container with ID starting with d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16 not found: ID does not exist" containerID="d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16" Sep 11 14:01:41.738061 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:41.738019 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16"} err="failed to get container status \"d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16\": rpc error: code = NotFound desc = could not find container \"d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16\": container with ID starting with d692f08af45f9f0a35f68ffec1efd9e42ec278ff2a74e19be5fc436f45139f16 not found: ID does not exist" Sep 11 14:01:41.738769 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:41.738741 2519 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7552-7754a4\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:42.002133 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.002040 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:42.103263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.103227 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:42.329814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.329741 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:42.517794 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.517762 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:42.586845 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.586819 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:42.641622 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.641586 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:01:42.642013 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.642001 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" containerName="test-curl-7552-7754a4" Sep 11 14:01:42.642055 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.642015 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" containerName="test-curl-7552-7754a4" Sep 11 14:01:42.642087 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.642083 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" containerName="test-curl-7552-7754a4" Sep 11 14:01:42.645991 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.645974 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.647907 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:42.647880 2519 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7552-7754a4\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:42.648607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.648568 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-30fbcdfa\"" Sep 11 14:01:42.648712 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.648568 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-30fbcdfa-dockercfg-95nnb\"" Sep 11 14:01:42.654288 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.654264 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:01:42.690846 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:01:42.690814 2519 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7552-7754a4\" is forbidden: User \"system:node:ip-10-0-139-201.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-201.ec2.internal' and this object" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" pod="opendatahub/test-curl-7552-7754a4" Sep 11 14:01:42.698436 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.698397 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.698608 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.698456 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrmh\" (UniqueName: \"kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.699811 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.699782 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4850dd70-bed8-4346-a23b-ab24bea28da9" path="/var/lib/kubelet/pods/4850dd70-bed8-4346-a23b-ab24bea28da9/volumes" Sep 11 14:01:42.710591 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.710555 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:01:42.713328 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.713308 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.724339 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.724316 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:01:42.799747 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.799700 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.799915 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.799764 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrmh\" (UniqueName: \"kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.799979 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.799911 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.800022 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.799990 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbl2\" (UniqueName: \"kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.800507 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.800486 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.812561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.812528 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrmh\" (UniqueName: \"kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh\") pod \"payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:42.900922 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.900832 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.900922 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.900895 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbl2\" (UniqueName: \"kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.901691 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.901669 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.909052 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.909022 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbl2\" (UniqueName: \"kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2\") pod \"payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:42.957030 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:42.956993 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:43.024292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.023947 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:43.085554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.085521 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:43.156275 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.156172 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:43.231632 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.231606 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:01:43.233225 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:43.233202 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fa2abb_db27_4510_9055_42e74d3c735f.slice/crio-879164bcbd722f70823d7fc71989806b65d6db65d8a6583fb0654baf246ab98c WatchSource:0}: Error finding container 879164bcbd722f70823d7fc71989806b65d6db65d8a6583fb0654baf246ab98c: Status 404 returned error can't find the container with id 879164bcbd722f70823d7fc71989806b65d6db65d8a6583fb0654baf246ab98c Sep 11 14:01:43.305712 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.305686 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:01:43.307662 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:43.307636 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da61d13_cbed_4569_9159_7efafb5b7d0a.slice/crio-93e84c750a117cf209cea0f2549490bed751fe0269641d48a3e519c07cc4a8c1 WatchSource:0}: Error finding container 93e84c750a117cf209cea0f2549490bed751fe0269641d48a3e519c07cc4a8c1: Status 404 returned error can't find the container with id 93e84c750a117cf209cea0f2549490bed751fe0269641d48a3e519c07cc4a8c1 Sep 11 14:01:43.681286 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.681244 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:43.742219 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.742176 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" event={"ID":"d2fa2abb-db27-4510-9055-42e74d3c735f","Type":"ContainerStarted","Data":"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20"} Sep 11 14:01:43.742424 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.742251 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" event={"ID":"d2fa2abb-db27-4510-9055-42e74d3c735f","Type":"ContainerStarted","Data":"879164bcbd722f70823d7fc71989806b65d6db65d8a6583fb0654baf246ab98c"} Sep 11 14:01:43.742424 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.742283 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:43.743790 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.743762 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" event={"ID":"1da61d13-cbed-4569-9159-7efafb5b7d0a","Type":"ContainerStarted","Data":"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8"} Sep 11 14:01:43.743914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.743812 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" event={"ID":"1da61d13-cbed-4569-9159-7efafb5b7d0a","Type":"ContainerStarted","Data":"93e84c750a117cf209cea0f2549490bed751fe0269641d48a3e519c07cc4a8c1"} Sep 11 14:01:43.743914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.743841 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:43.759275 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.759204 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" podStartSLOduration=1.759183986 podStartE2EDuration="1.759183986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:43.757777692 +0000 UTC m=+771.512092583" watchObservedRunningTime="2026-09-11 14:01:43.759183986 +0000 UTC m=+771.513498878" Sep 11 14:01:43.766498 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.766474 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:43.779040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.778981 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" podStartSLOduration=1.778959197 podStartE2EDuration="1.778959197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:43.774553679 +0000 UTC m=+771.528868571" watchObservedRunningTime="2026-09-11 14:01:43.778959197 +0000 UTC m=+771.533274091" Sep 11 14:01:43.811359 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.811324 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:01:43.814823 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.814801 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:43.818725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.818704 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-498d98-dockercfg-cstnc\"" Sep 11 14:01:43.819005 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.818987 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-498d98\"" Sep 11 14:01:43.827563 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.827539 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:01:43.867204 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.867165 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:01:43.871599 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.871552 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:43.882419 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.882378 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:01:43.912015 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.911980 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:43.912191 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:43.912041 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.013219 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.013113 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.013219 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.013180 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.013459 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.013245 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.013459 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.013279 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmcw\" (UniqueName: \"kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.013976 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.013954 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.020948 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.020919 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq\") pod \"payload-processing-e2e-autores-498d98-658857b485-ksp5f\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.114123 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.114083 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.114517 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.114172 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmcw\" (UniqueName: \"kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.114824 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.114795 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.121958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.121934 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmcw\" (UniqueName: \"kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw\") pod \"payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.134780 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.134757 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.186788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.185538 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.388114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.388076 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:44.424621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.424593 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:01:44.425543 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:44.425518 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338457c5_d7da_4621_ad5f_efccf2742a75.slice/crio-81a2931e2d22b09b0ed29d0992abfd75632f80a17effdc0c7395b406dcb582ad WatchSource:0}: Error finding container 81a2931e2d22b09b0ed29d0992abfd75632f80a17effdc0c7395b406dcb582ad: Status 404 returned error can't find the container with id 81a2931e2d22b09b0ed29d0992abfd75632f80a17effdc0c7395b406dcb582ad Sep 11 14:01:44.466652 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.465968 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:01:44.469847 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:01:44.469798 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195957f5_559e_4b5c_9ada_f9b29fd31e66.slice/crio-70fa1513ed7031e024248d95b9c54679947e5b175ef19cfe44401fb2d6dc2daf WatchSource:0}: Error finding container 70fa1513ed7031e024248d95b9c54679947e5b175ef19cfe44401fb2d6dc2daf: Status 404 returned error can't find the container with id 70fa1513ed7031e024248d95b9c54679947e5b175ef19cfe44401fb2d6dc2daf Sep 11 14:01:44.753001 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.752887 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" event={"ID":"338457c5-d7da-4621-ad5f-efccf2742a75","Type":"ContainerStarted","Data":"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15"} Sep 11 14:01:44.753001 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.752957 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" event={"ID":"338457c5-d7da-4621-ad5f-efccf2742a75","Type":"ContainerStarted","Data":"81a2931e2d22b09b0ed29d0992abfd75632f80a17effdc0c7395b406dcb582ad"} Sep 11 14:01:44.753001 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.752988 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:44.755002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.754972 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" event={"ID":"195957f5-559e-4b5c-9ada-f9b29fd31e66","Type":"ContainerStarted","Data":"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47"} Sep 11 14:01:44.755163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.755024 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" event={"ID":"195957f5-559e-4b5c-9ada-f9b29fd31e66","Type":"ContainerStarted","Data":"70fa1513ed7031e024248d95b9c54679947e5b175ef19cfe44401fb2d6dc2daf"} Sep 11 14:01:44.755163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.755127 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:44.768825 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.768766 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" podStartSLOduration=1.768743849 podStartE2EDuration="1.768743849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:44.767515249 +0000 UTC m=+772.521830141" watchObservedRunningTime="2026-09-11 14:01:44.768743849 +0000 UTC m=+772.523058741" Sep 11 14:01:44.785706 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:44.785645 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" podStartSLOduration=1.785623092 podStartE2EDuration="1.785623092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:44.784972271 +0000 UTC m=+772.539287163" watchObservedRunningTime="2026-09-11 14:01:44.785623092 +0000 UTC m=+772.539938004" Sep 11 14:01:45.077285 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:45.077199 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:47.701945 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:47.701910 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:01:48.448798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:48.448765 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:48.624653 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:48.624615 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:48.707451 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:48.707377 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:01:48.716607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:48.716560 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:48.817429 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:48.817384 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:51.553950 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:51.553911 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:53.594103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:53.594049 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:54.757063 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:54.757031 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:01:54.757483 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:54.757282 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:01:55.019901 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:55.019789 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:55.147268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:55.147234 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:55.304723 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:55.304635 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:55.763252 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:55.763220 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:01:55.763645 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:55.763290 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:01:56.931409 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:56.931377 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:58.303155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:58.303123 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:58.691677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:58.691648 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:58.937316 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:58.937285 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:59.189162 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:01:59.189129 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:01.090633 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:01.090600 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:01.253672 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:01.253639 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:01.354103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:01.354030 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:01.541878 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:01.541843 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:02.589334 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:02.589298 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:02.689804 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:02.689764 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:03.058721 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:03.058568 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:03.968092 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:03.968058 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:04.066692 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.066657 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:04.166220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.166188 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:04.499636 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.499545 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:04.599537 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.599505 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:04.639044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.639016 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:02:04.642959 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.642940 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.645356 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.645336 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-da9534\"" Sep 11 14:02:04.645459 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.645394 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-da9534-dockercfg-d7nn7\"" Sep 11 14:02:04.656515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.656492 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:02:04.712789 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.712761 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:02:04.716317 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.716298 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.722664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.722643 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.722912 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.722794 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zksl\" (UniqueName: \"kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.790016 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.789949 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:02:04.823502 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.823465 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.823676 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.823510 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bw7\" (UniqueName: \"kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.823676 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.823540 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.823676 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.823663 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zksl\" (UniqueName: \"kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.824251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.824233 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.837915 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.837893 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zksl\" (UniqueName: \"kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl\") pod \"payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:04.924507 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.924471 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.924710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.924520 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bw7\" (UniqueName: \"kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.925202 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.925181 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.933422 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.933403 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bw7\" (UniqueName: \"kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7\") pod \"payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:04.953497 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:04.953473 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:05.026007 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.025971 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:05.147274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.147247 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:05.228979 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.228953 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:02:05.230624 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:05.230571 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2282259a_a663_4e86_b50f_80edfc7e5c03.slice/crio-85b7ad58b905253eb7938802ac4b2564c65c3287a83d725451ce57f16d0d1872 WatchSource:0}: Error finding container 85b7ad58b905253eb7938802ac4b2564c65c3287a83d725451ce57f16d0d1872: Status 404 returned error can't find the container with id 85b7ad58b905253eb7938802ac4b2564c65c3287a83d725451ce57f16d0d1872 Sep 11 14:02:05.309775 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.309748 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:02:05.311311 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:05.311263 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c6b2e7_e755_4817_9284_606df07c2887.slice/crio-a77b6c44ad541f9d1fa71bb0299842e46b5d41408dc1c45681bbe7a65e9c9fb3 WatchSource:0}: Error finding container a77b6c44ad541f9d1fa71bb0299842e46b5d41408dc1c45681bbe7a65e9c9fb3: Status 404 returned error can't find the container with id a77b6c44ad541f9d1fa71bb0299842e46b5d41408dc1c45681bbe7a65e9c9fb3 Sep 11 14:02:05.881168 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.881115 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" event={"ID":"2282259a-a663-4e86-b50f-80edfc7e5c03","Type":"ContainerStarted","Data":"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75"} Sep 11 14:02:05.881368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.881186 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:05.881368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.881209 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" event={"ID":"2282259a-a663-4e86-b50f-80edfc7e5c03","Type":"ContainerStarted","Data":"85b7ad58b905253eb7938802ac4b2564c65c3287a83d725451ce57f16d0d1872"} Sep 11 14:02:05.882942 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.882917 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" event={"ID":"21c6b2e7-e755-4817-9284-606df07c2887","Type":"ContainerStarted","Data":"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934"} Sep 11 14:02:05.883044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.882962 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" event={"ID":"21c6b2e7-e755-4817-9284-606df07c2887","Type":"ContainerStarted","Data":"a77b6c44ad541f9d1fa71bb0299842e46b5d41408dc1c45681bbe7a65e9c9fb3"} Sep 11 14:02:05.883044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.882990 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:05.899596 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.899527 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" podStartSLOduration=1.899510551 podStartE2EDuration="1.899510551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:05.896808492 +0000 UTC m=+793.651123384" watchObservedRunningTime="2026-09-11 14:02:05.899510551 +0000 UTC m=+793.653825443" Sep 11 14:02:05.912794 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:05.912745 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" podStartSLOduration=1.912723924 podStartE2EDuration="1.912723924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:05.91133172 +0000 UTC m=+793.665646611" watchObservedRunningTime="2026-09-11 14:02:05.912723924 +0000 UTC m=+793.667038816" Sep 11 14:02:06.065531 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.065499 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:06.407520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.407478 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz"] Sep 11 14:02:06.411358 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.411336 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:06.413534 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.413513 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:02:06.417934 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.417909 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz"] Sep 11 14:02:06.442801 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.442770 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz\" (UID: \"68dbe286-d662-47c5-83e3-2cf1e76ae985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:06.543688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.543646 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz\" (UID: \"68dbe286-d662-47c5-83e3-2cf1e76ae985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:06.544346 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.544327 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz\" (UID: \"68dbe286-d662-47c5-83e3-2cf1e76ae985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:06.723227 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.723145 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:06.961097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.961065 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:06.997402 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:06.997379 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz"] Sep 11 14:02:06.999449 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:06.999421 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dbe286_d662_47c5_83e3_2cf1e76ae985.slice/crio-debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516 WatchSource:0}: Error finding container debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516: Status 404 returned error can't find the container with id debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516 Sep 11 14:02:07.817246 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:07.817209 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:07.895113 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:07.895081 2519 generic.go:356] "Generic (PLEG): container finished" podID="68dbe286-d662-47c5-83e3-2cf1e76ae985" containerID="550c897b0873315a243c5b8746eac3581d56759b42ec4c60eac6dfab14352b95" exitCode=0 Sep 11 14:02:07.895329 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:07.895172 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" event={"ID":"68dbe286-d662-47c5-83e3-2cf1e76ae985","Type":"ContainerDied","Data":"550c897b0873315a243c5b8746eac3581d56759b42ec4c60eac6dfab14352b95"} Sep 11 14:02:07.895329 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:07.895222 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" event={"ID":"68dbe286-d662-47c5-83e3-2cf1e76ae985","Type":"ContainerStarted","Data":"debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516"} Sep 11 14:02:08.027827 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:08.027736 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:08.128273 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:08.128241 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:08.739136 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:08.739100 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:09.034396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.034373 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:09.070700 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.070667 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca\") pod \"68dbe286-d662-47c5-83e3-2cf1e76ae985\" (UID: \"68dbe286-d662-47c5-83e3-2cf1e76ae985\") " Sep 11 14:02:09.071073 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.071044 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68dbe286-d662-47c5-83e3-2cf1e76ae985" (UID: "68dbe286-d662-47c5-83e3-2cf1e76ae985"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:02:09.172063 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.172031 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dbe286-d662-47c5-83e3-2cf1e76ae985-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:02:09.907523 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.907494 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" Sep 11 14:02:09.907726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.907527 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz" event={"ID":"68dbe286-d662-47c5-83e3-2cf1e76ae985","Type":"ContainerDied","Data":"debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516"} Sep 11 14:02:09.907726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:09.907600 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="debdc20d1b4ecb6cbb5587814e17def57f2ac72e1b9b5ec482caf30bd2693516" Sep 11 14:02:11.486548 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:11.486513 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:12.181505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:12.181473 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:12.944191 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:12.944162 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.048018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.047979 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:02:13.048287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.048253 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" podUID="1da61d13-cbed-4569-9159-7efafb5b7d0a" containerName="payload-pre-processing" containerID="cri-o://fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8" gracePeriod=30 Sep 11 14:02:13.056759 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.056429 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:02:13.056759 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.056738 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" podUID="d2fa2abb-db27-4510-9055-42e74d3c735f" containerName="payload-processing" containerID="cri-o://3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20" gracePeriod=30 Sep 11 14:02:13.114234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.114190 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.217830 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.217486 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.269996 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.269965 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz"] Sep 11 14:02:13.276904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.276876 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30fbcdfa-2vjtz"] Sep 11 14:02:13.285344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.285292 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.335289 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.335266 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:02:13.338042 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.338023 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:02:13.416769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.416731 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume\") pod \"d2fa2abb-db27-4510-9055-42e74d3c735f\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " Sep 11 14:02:13.416965 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.416882 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xbl2\" (UniqueName: \"kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2\") pod \"1da61d13-cbed-4569-9159-7efafb5b7d0a\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " Sep 11 14:02:13.417030 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.416964 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume\") pod \"1da61d13-cbed-4569-9159-7efafb5b7d0a\" (UID: \"1da61d13-cbed-4569-9159-7efafb5b7d0a\") " Sep 11 14:02:13.417030 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.416996 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lrmh\" (UniqueName: \"kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh\") pod \"d2fa2abb-db27-4510-9055-42e74d3c735f\" (UID: \"d2fa2abb-db27-4510-9055-42e74d3c735f\") " Sep 11 14:02:13.417370 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.417339 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2fa2abb-db27-4510-9055-42e74d3c735f" (UID: "d2fa2abb-db27-4510-9055-42e74d3c735f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:02:13.417447 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.417399 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1da61d13-cbed-4569-9159-7efafb5b7d0a" (UID: "1da61d13-cbed-4569-9159-7efafb5b7d0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:02:13.419962 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.419926 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh" (OuterVolumeSpecName: "kube-api-access-5lrmh") pod "d2fa2abb-db27-4510-9055-42e74d3c735f" (UID: "d2fa2abb-db27-4510-9055-42e74d3c735f"). InnerVolumeSpecName "kube-api-access-5lrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:02:13.420078 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.420023 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2" (OuterVolumeSpecName: "kube-api-access-9xbl2") pod "1da61d13-cbed-4569-9159-7efafb5b7d0a" (UID: "1da61d13-cbed-4569-9159-7efafb5b7d0a"). InnerVolumeSpecName "kube-api-access-9xbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:02:13.433239 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.433212 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.517974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.517890 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrmh\" (UniqueName: \"kubernetes.io/projected/d2fa2abb-db27-4510-9055-42e74d3c735f-kube-api-access-5lrmh\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:02:13.517974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.517919 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fa2abb-db27-4510-9055-42e74d3c735f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:02:13.517974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.517935 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbl2\" (UniqueName: \"kubernetes.io/projected/1da61d13-cbed-4569-9159-7efafb5b7d0a-kube-api-access-9xbl2\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:02:13.517974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.517949 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da61d13-cbed-4569-9159-7efafb5b7d0a-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:02:13.931121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.931085 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:13.934526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.934496 2519 generic.go:356] "Generic (PLEG): container finished" podID="d2fa2abb-db27-4510-9055-42e74d3c735f" containerID="3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20" exitCode=0 Sep 11 14:02:13.934668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.934538 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" Sep 11 14:02:13.934668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.934600 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" event={"ID":"d2fa2abb-db27-4510-9055-42e74d3c735f","Type":"ContainerDied","Data":"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20"} Sep 11 14:02:13.934668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.934650 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs" event={"ID":"d2fa2abb-db27-4510-9055-42e74d3c735f","Type":"ContainerDied","Data":"879164bcbd722f70823d7fc71989806b65d6db65d8a6583fb0654baf246ab98c"} Sep 11 14:02:13.934821 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.934672 2519 scope.go:117] "RemoveContainer" containerID="3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20" Sep 11 14:02:13.935984 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.935961 2519 generic.go:356] "Generic (PLEG): container finished" podID="1da61d13-cbed-4569-9159-7efafb5b7d0a" containerID="fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8" exitCode=0 Sep 11 14:02:13.936096 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.936005 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" Sep 11 14:02:13.936096 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.936038 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" event={"ID":"1da61d13-cbed-4569-9159-7efafb5b7d0a","Type":"ContainerDied","Data":"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8"} Sep 11 14:02:13.936096 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.936085 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s" event={"ID":"1da61d13-cbed-4569-9159-7efafb5b7d0a","Type":"ContainerDied","Data":"93e84c750a117cf209cea0f2549490bed751fe0269641d48a3e519c07cc4a8c1"} Sep 11 14:02:13.945844 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.945764 2519 scope.go:117] "RemoveContainer" containerID="3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20" Sep 11 14:02:13.946165 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:02:13.946116 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20\": container with ID starting with 3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20 not found: ID does not exist" containerID="3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20" Sep 11 14:02:13.946165 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.946148 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20"} err="failed to get container status \"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20\": rpc error: code = NotFound desc = could not find container \"3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20\": container with ID starting with 3f0071cf5f320b9e95bf9a5442c48a088abb70afee5b7846a170dc7e767dfe20 not found: ID does not exist" Sep 11 14:02:13.946262 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.946171 2519 scope.go:117] "RemoveContainer" containerID="fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8" Sep 11 14:02:13.954438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.954409 2519 scope.go:117] "RemoveContainer" containerID="fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8" Sep 11 14:02:13.954731 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:02:13.954713 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8\": container with ID starting with fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8 not found: ID does not exist" containerID="fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8" Sep 11 14:02:13.954791 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.954737 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8"} err="failed to get container status \"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8\": rpc error: code = NotFound desc = could not find container \"fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8\": container with ID starting with fa542a5e2ea57fb893a9f0c1ef05982c01f4ed949766a9e0aa804745cd1b25e8 not found: ID does not exist" Sep 11 14:02:13.958374 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.958353 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:02:13.962922 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.962902 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30fbcdfa-5f764dbd9c-ptgqs"] Sep 11 14:02:13.974293 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.974272 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:02:13.978082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:13.978060 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-30fbcdfa-c747fcc9d-4t54s"] Sep 11 14:02:14.016063 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.016037 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:14.132006 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.131971 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:14.445727 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.445694 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:14.692559 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.692527 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da61d13-cbed-4569-9159-7efafb5b7d0a" path="/var/lib/kubelet/pods/1da61d13-cbed-4569-9159-7efafb5b7d0a/volumes" Sep 11 14:02:14.692905 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.692892 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68dbe286-d662-47c5-83e3-2cf1e76ae985" path="/var/lib/kubelet/pods/68dbe286-d662-47c5-83e3-2cf1e76ae985/volumes" Sep 11 14:02:14.693217 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.693205 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2fa2abb-db27-4510-9055-42e74d3c735f" path="/var/lib/kubelet/pods/d2fa2abb-db27-4510-9055-42e74d3c735f/volumes" Sep 11 14:02:14.699479 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:14.699425 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:16.890797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:16.890766 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:02:16.891274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:16.890941 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:02:20.343469 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:20.343440 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:20.442779 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:20.442747 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:20.551726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:20.551694 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:20.879569 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:20.879524 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:21.038500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.038464 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:21.140184 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.140103 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:21.170104 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170064 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:02:21.170539 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170524 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da61d13-cbed-4569-9159-7efafb5b7d0a" containerName="payload-pre-processing" Sep 11 14:02:21.170614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170543 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da61d13-cbed-4569-9159-7efafb5b7d0a" containerName="payload-pre-processing" Sep 11 14:02:21.170614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170556 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fa2abb-db27-4510-9055-42e74d3c735f" containerName="payload-processing" Sep 11 14:02:21.170614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170562 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fa2abb-db27-4510-9055-42e74d3c735f" containerName="payload-processing" Sep 11 14:02:21.170614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170587 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68dbe286-d662-47c5-83e3-2cf1e76ae985" containerName="revoke-keys" Sep 11 14:02:21.170614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170593 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="68dbe286-d662-47c5-83e3-2cf1e76ae985" containerName="revoke-keys" Sep 11 14:02:21.170769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170681 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da61d13-cbed-4569-9159-7efafb5b7d0a" containerName="payload-pre-processing" Sep 11 14:02:21.170769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170692 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fa2abb-db27-4510-9055-42e74d3c735f" containerName="payload-processing" Sep 11 14:02:21.170769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.170704 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="68dbe286-d662-47c5-83e3-2cf1e76ae985" containerName="revoke-keys" Sep 11 14:02:21.173606 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.173564 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.175915 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.175896 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0d4b8af1\"" Sep 11 14:02:21.176018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.175898 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0d4b8af1-dockercfg-x9hp7\"" Sep 11 14:02:21.193089 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.193058 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:02:21.226074 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.226043 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:02:21.229955 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.229934 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.239549 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.239524 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:02:21.300669 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.300635 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.300831 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.300703 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmjl\" (UniqueName: \"kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.401976 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.401892 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.401976 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.401967 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.402411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.402014 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmjl\" (UniqueName: \"kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.402411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.402046 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdhp\" (UniqueName: \"kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.402772 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.402750 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.411538 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.411510 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmjl\" (UniqueName: \"kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl\") pod \"payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.484222 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.484182 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.503605 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.503557 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.503771 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.503752 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdhp\" (UniqueName: \"kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.504263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.504244 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.512042 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.512012 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdhp\" (UniqueName: \"kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp\") pod \"payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.543543 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.542947 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.756852 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.756823 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:02:21.758206 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:21.758182 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053c964f_db70_4fe1_8dda_4619a0b41518.slice/crio-2b296f342d5c429c43972ef83950ab231cb941305a74a69687560e9fd79ba47b WatchSource:0}: Error finding container 2b296f342d5c429c43972ef83950ab231cb941305a74a69687560e9fd79ba47b: Status 404 returned error can't find the container with id 2b296f342d5c429c43972ef83950ab231cb941305a74a69687560e9fd79ba47b Sep 11 14:02:21.822271 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.822249 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:02:21.824444 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:21.824420 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81962592_2c2b_4e86_b759_68374476f3af.slice/crio-a538b6979be22d4719683eb10394791f743f0d763ca3e844b62056c864e904f4 WatchSource:0}: Error finding container a538b6979be22d4719683eb10394791f743f0d763ca3e844b62056c864e904f4: Status 404 returned error can't find the container with id a538b6979be22d4719683eb10394791f743f0d763ca3e844b62056c864e904f4 Sep 11 14:02:21.986196 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.986154 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" event={"ID":"053c964f-db70-4fe1-8dda-4619a0b41518","Type":"ContainerStarted","Data":"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452"} Sep 11 14:02:21.986391 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.986227 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:21.986391 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.986254 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" event={"ID":"053c964f-db70-4fe1-8dda-4619a0b41518","Type":"ContainerStarted","Data":"2b296f342d5c429c43972ef83950ab231cb941305a74a69687560e9fd79ba47b"} Sep 11 14:02:21.988103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.988070 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" event={"ID":"81962592-2c2b-4e86-b759-68374476f3af","Type":"ContainerStarted","Data":"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83"} Sep 11 14:02:21.988244 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.988137 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:21.988244 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:21.988159 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" event={"ID":"81962592-2c2b-4e86-b759-68374476f3af","Type":"ContainerStarted","Data":"a538b6979be22d4719683eb10394791f743f0d763ca3e844b62056c864e904f4"} Sep 11 14:02:22.004303 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:22.004251 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" podStartSLOduration=1.004236529 podStartE2EDuration="1.004236529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:22.002051928 +0000 UTC m=+809.756366819" watchObservedRunningTime="2026-09-11 14:02:22.004236529 +0000 UTC m=+809.758551464" Sep 11 14:02:22.018981 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:22.018883 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" podStartSLOduration=1.018864296 podStartE2EDuration="1.018864296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:22.01728747 +0000 UTC m=+809.771602363" watchObservedRunningTime="2026-09-11 14:02:22.018864296 +0000 UTC m=+809.773179186" Sep 11 14:02:25.177911 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:25.177866 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:27.567627 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:27.567554 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:32.996658 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:32.996625 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:02:32.997053 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:32.996693 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:02:41.488215 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:41.488175 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:45.352261 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:45.352189 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:53.841524 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:53.841493 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:56.108929 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:56.108896 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:58.359099 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.359063 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:02:58.363186 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.363165 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.366119 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.366092 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-498d98\"/\"kube-root-ca.crt\"" Sep 11 14:02:58.366119 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.366109 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-498d98\"/\"default-dockercfg-6ltcz\"" Sep 11 14:02:58.366321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.366164 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-498d98\"/\"openshift-service-ca.crt\"" Sep 11 14:02:58.366321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.366172 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-498d98\"/\"autores-model-498d98-kserve-self-signed-certs\"" Sep 11 14:02:58.373564 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.373545 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:02:58.380282 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.380257 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:58.480308 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480272 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.480529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480328 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.480529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480356 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.480529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480385 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.480529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480487 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.480529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.480520 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwd5l\" (UniqueName: \"kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581331 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581289 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581337 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581458 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581505 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581727 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581600 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581727 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581641 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwd5l\" (UniqueName: \"kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581836 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581749 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.581836 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581823 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.582404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.581919 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.590843 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.590800 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.591426 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.591407 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.593224 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.593202 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwd5l\" (UniqueName: \"kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l\") pod \"autores-model-498d98-kserve-6464594d4c-6ss6f\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.677131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.677046 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:02:58.954706 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:58.954682 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:02:58.957600 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:02:58.957553 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d8d64a9_0a30_43c2_b614_f3950bd745b4.slice/crio-3f655f4e4a21ad405917513c24c43374050440b7d228eb2514b010d59141b128 WatchSource:0}: Error finding container 3f655f4e4a21ad405917513c24c43374050440b7d228eb2514b010d59141b128: Status 404 returned error can't find the container with id 3f655f4e4a21ad405917513c24c43374050440b7d228eb2514b010d59141b128 Sep 11 14:02:59.196759 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:59.196727 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" event={"ID":"4d8d64a9-0a30-43c2-b614-f3950bd745b4","Type":"ContainerStarted","Data":"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e"} Sep 11 14:02:59.196945 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:59.196769 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" event={"ID":"4d8d64a9-0a30-43c2-b614-f3950bd745b4","Type":"ContainerStarted","Data":"3f655f4e4a21ad405917513c24c43374050440b7d228eb2514b010d59141b128"} Sep 11 14:02:59.217037 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:02:59.216927 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" podStartSLOduration=1.216912265 podStartE2EDuration="1.216912265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:59.215795954 +0000 UTC m=+846.970110848" watchObservedRunningTime="2026-09-11 14:02:59.216912265 +0000 UTC m=+846.971227195" Sep 11 14:03:01.464719 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:01.464674 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.117349 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.117316 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.137305 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.137270 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:03:02.137542 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.137513 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" podUID="053c964f-db70-4fe1-8dda-4619a0b41518" containerName="payload-processing" containerID="cri-o://c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452" gracePeriod=30 Sep 11 14:03:02.145699 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.145675 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:03:02.146086 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.146057 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" podUID="81962592-2c2b-4e86-b759-68374476f3af" containerName="payload-pre-processing" containerID="cri-o://0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83" gracePeriod=30 Sep 11 14:03:02.217837 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.217802 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.318744 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.318685 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.416792 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.416564 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:03:02.418543 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.418518 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.420141 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.420125 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:03:02.489631 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.489552 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:02.519175 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519148 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume\") pod \"81962592-2c2b-4e86-b759-68374476f3af\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " Sep 11 14:03:02.519323 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519282 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xmjl\" (UniqueName: \"kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl\") pod \"053c964f-db70-4fe1-8dda-4619a0b41518\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " Sep 11 14:03:02.519369 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519321 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhdhp\" (UniqueName: \"kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp\") pod \"81962592-2c2b-4e86-b759-68374476f3af\" (UID: \"81962592-2c2b-4e86-b759-68374476f3af\") " Sep 11 14:03:02.519410 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519393 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume\") pod \"053c964f-db70-4fe1-8dda-4619a0b41518\" (UID: \"053c964f-db70-4fe1-8dda-4619a0b41518\") " Sep 11 14:03:02.519634 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519569 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81962592-2c2b-4e86-b759-68374476f3af" (UID: "81962592-2c2b-4e86-b759-68374476f3af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:02.519810 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519795 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81962592-2c2b-4e86-b759-68374476f3af-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:02.519882 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.519814 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "053c964f-db70-4fe1-8dda-4619a0b41518" (UID: "053c964f-db70-4fe1-8dda-4619a0b41518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:02.521365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.521338 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl" (OuterVolumeSpecName: "kube-api-access-4xmjl") pod "053c964f-db70-4fe1-8dda-4619a0b41518" (UID: "053c964f-db70-4fe1-8dda-4619a0b41518"). InnerVolumeSpecName "kube-api-access-4xmjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:02.521523 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.521501 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp" (OuterVolumeSpecName: "kube-api-access-rhdhp") pod "81962592-2c2b-4e86-b759-68374476f3af" (UID: "81962592-2c2b-4e86-b759-68374476f3af"). InnerVolumeSpecName "kube-api-access-rhdhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:02.621085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.621054 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmjl\" (UniqueName: \"kubernetes.io/projected/053c964f-db70-4fe1-8dda-4619a0b41518-kube-api-access-4xmjl\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:02.621085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.621081 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdhp\" (UniqueName: \"kubernetes.io/projected/81962592-2c2b-4e86-b759-68374476f3af-kube-api-access-rhdhp\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:02.621085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.621090 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/053c964f-db70-4fe1-8dda-4619a0b41518-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:02.918591 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:02.918545 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:03.194172 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.194143 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:03.221683 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.221650 2519 generic.go:356] "Generic (PLEG): container finished" podID="053c964f-db70-4fe1-8dda-4619a0b41518" containerID="c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452" exitCode=0 Sep 11 14:03:03.221854 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.221714 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" Sep 11 14:03:03.221854 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.221738 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" event={"ID":"053c964f-db70-4fe1-8dda-4619a0b41518","Type":"ContainerDied","Data":"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452"} Sep 11 14:03:03.221854 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.221832 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9" event={"ID":"053c964f-db70-4fe1-8dda-4619a0b41518","Type":"ContainerDied","Data":"2b296f342d5c429c43972ef83950ab231cb941305a74a69687560e9fd79ba47b"} Sep 11 14:03:03.222037 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.221877 2519 scope.go:117] "RemoveContainer" containerID="c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452" Sep 11 14:03:03.223279 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.223173 2519 generic.go:356] "Generic (PLEG): container finished" podID="81962592-2c2b-4e86-b759-68374476f3af" containerID="0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83" exitCode=0 Sep 11 14:03:03.223279 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.223231 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" Sep 11 14:03:03.223279 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.223265 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" event={"ID":"81962592-2c2b-4e86-b759-68374476f3af","Type":"ContainerDied","Data":"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83"} Sep 11 14:03:03.223425 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.223319 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw" event={"ID":"81962592-2c2b-4e86-b759-68374476f3af","Type":"ContainerDied","Data":"a538b6979be22d4719683eb10394791f743f0d763ca3e844b62056c864e904f4"} Sep 11 14:03:03.230745 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.230731 2519 scope.go:117] "RemoveContainer" containerID="c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452" Sep 11 14:03:03.231032 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:03.231010 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452\": container with ID starting with c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452 not found: ID does not exist" containerID="c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452" Sep 11 14:03:03.231084 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.231039 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452"} err="failed to get container status \"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452\": rpc error: code = NotFound desc = could not find container \"c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452\": container with ID starting with c997741b8a15e51f69d0dfe8faead26ba899c8448a6d7b29838b004dd1926452 not found: ID does not exist" Sep 11 14:03:03.231084 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.231055 2519 scope.go:117] "RemoveContainer" containerID="0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83" Sep 11 14:03:03.239212 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.239197 2519 scope.go:117] "RemoveContainer" containerID="0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83" Sep 11 14:03:03.239485 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:03.239467 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83\": container with ID starting with 0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83 not found: ID does not exist" containerID="0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83" Sep 11 14:03:03.239589 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.239489 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83"} err="failed to get container status \"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83\": rpc error: code = NotFound desc = could not find container \"0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83\": container with ID starting with 0a25cabae881e6b6c3995583121ad97fdc65e90b298058c8e735364383e6db83 not found: ID does not exist" Sep 11 14:03:03.241065 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.241043 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:03:03.243225 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.243205 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0d4b8af1-c7cb787d-hchjw"] Sep 11 14:03:03.252333 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.252312 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:03:03.260912 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.260888 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0d4b8af1-8ff5d5c98-krhk9"] Sep 11 14:03:03.407813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:03.407774 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:04.692281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:04.692251 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053c964f-db70-4fe1-8dda-4619a0b41518" path="/var/lib/kubelet/pods/053c964f-db70-4fe1-8dda-4619a0b41518/volumes" Sep 11 14:03:04.692684 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:04.692670 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81962592-2c2b-4e86-b759-68374476f3af" path="/var/lib/kubelet/pods/81962592-2c2b-4e86-b759-68374476f3af/volumes" Sep 11 14:03:05.385615 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:05.385570 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:05.474431 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:05.474393 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:06.658796 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:06.658766 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:06.761407 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:06.761375 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:06.821788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:06.821755 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:07.194157 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.194128 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:07.255569 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.255536 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:07.256090 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256073 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053c964f-db70-4fe1-8dda-4619a0b41518" containerName="payload-processing" Sep 11 14:03:07.256145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256092 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="053c964f-db70-4fe1-8dda-4619a0b41518" containerName="payload-processing" Sep 11 14:03:07.256145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256110 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81962592-2c2b-4e86-b759-68374476f3af" containerName="payload-pre-processing" Sep 11 14:03:07.256145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256116 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="81962592-2c2b-4e86-b759-68374476f3af" containerName="payload-pre-processing" Sep 11 14:03:07.256263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256221 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="053c964f-db70-4fe1-8dda-4619a0b41518" containerName="payload-processing" Sep 11 14:03:07.256263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.256236 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="81962592-2c2b-4e86-b759-68374476f3af" containerName="payload-pre-processing" Sep 11 14:03:07.261226 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.261205 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.263360 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.263341 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bd7f15a9\"" Sep 11 14:03:07.270845 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.270824 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:07.276782 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.276754 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:07.284762 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.284734 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:07.288752 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.288728 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.290880 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.290859 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bd7f15a9-dockercfg-d7wtv\"" Sep 11 14:03:07.300885 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.300857 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:07.365933 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.365896 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.366097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.365943 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.366097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.365977 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh47\" (UniqueName: \"kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.366097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.366065 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgb9j\" (UniqueName: \"kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.467404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.467305 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgb9j\" (UniqueName: \"kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.467618 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.467404 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.467618 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.467449 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.467618 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.467494 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh47\" (UniqueName: \"kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.468205 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.468182 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.468394 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.468371 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.476078 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.476047 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgb9j\" (UniqueName: \"kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j\") pod \"payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.477331 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.477305 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh47\" (UniqueName: \"kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47\") pod \"payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.574909 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.574860 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:07.603237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.603201 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:07.862018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.861985 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:07.863402 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:07.863373 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da05bff_9a27_40a7_bbe6_7c8754dbd9bb.slice/crio-e6cd33917219ee8f75a800adc8658e1a72fec336344aa36031d2a48884fb835f WatchSource:0}: Error finding container e6cd33917219ee8f75a800adc8658e1a72fec336344aa36031d2a48884fb835f: Status 404 returned error can't find the container with id e6cd33917219ee8f75a800adc8658e1a72fec336344aa36031d2a48884fb835f Sep 11 14:03:07.900540 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:07.900514 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:07.902557 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:07.902533 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee304fc_dd8f_4f59_ae19_b23fe464a177.slice/crio-a1a9db4e0aa8bf96f0d4062465210b3a4fc3b5b1655a66b810284752552fcbfb WatchSource:0}: Error finding container a1a9db4e0aa8bf96f0d4062465210b3a4fc3b5b1655a66b810284752552fcbfb: Status 404 returned error can't find the container with id a1a9db4e0aa8bf96f0d4062465210b3a4fc3b5b1655a66b810284752552fcbfb Sep 11 14:03:08.255539 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.255444 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" event={"ID":"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb","Type":"ContainerStarted","Data":"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3"} Sep 11 14:03:08.255539 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.255496 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" event={"ID":"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb","Type":"ContainerStarted","Data":"e6cd33917219ee8f75a800adc8658e1a72fec336344aa36031d2a48884fb835f"} Sep 11 14:03:08.255795 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.255550 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:08.257187 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.257159 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" event={"ID":"7ee304fc-dd8f-4f59-ae19-b23fe464a177","Type":"ContainerStarted","Data":"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9"} Sep 11 14:03:08.257305 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.257221 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" event={"ID":"7ee304fc-dd8f-4f59-ae19-b23fe464a177","Type":"ContainerStarted","Data":"a1a9db4e0aa8bf96f0d4062465210b3a4fc3b5b1655a66b810284752552fcbfb"} Sep 11 14:03:08.257305 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.257252 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:08.271534 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.271490 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" podStartSLOduration=1.271471056 podStartE2EDuration="1.271471056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:08.270216344 +0000 UTC m=+856.024531235" watchObservedRunningTime="2026-09-11 14:03:08.271471056 +0000 UTC m=+856.025785949" Sep 11 14:03:08.287699 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.287653 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" podStartSLOduration=1.287635824 podStartE2EDuration="1.287635824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:08.285120464 +0000 UTC m=+856.039435355" watchObservedRunningTime="2026-09-11 14:03:08.287635824 +0000 UTC m=+856.041950714" Sep 11 14:03:08.677992 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.677956 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:08.678173 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.678013 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:08.696372 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:08.696339 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:09.274471 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:09.274445 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:19.190155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:19.190100 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:19.265787 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:19.265752 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:19.265955 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:19.265830 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:21.479535 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:21.479507 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:24.740548 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:24.740508 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:03:24.758297 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:24.758263 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:24.979661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:24.979629 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:27.679182 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:27.679139 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:28.873284 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:28.873255 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:28.934589 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:28.934545 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.045241 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.045202 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.672047 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.672012 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.701162 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.701130 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:03:35.701408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.701381 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" containerName="payload-processing" containerID="cri-o://b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15" gracePeriod=30 Sep 11 14:03:35.709438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.709405 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:03:35.710371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.709848 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerName="payload-pre-processing" containerID="cri-o://842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47" gracePeriod=30 Sep 11 14:03:35.745773 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.745738 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.762472 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.762430 2519 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.52:9004: connect: connection refused" Sep 11 14:03:35.762657 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.762440 2519 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.53:9004: connect: connection refused" Sep 11 14:03:35.808637 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.808566 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.885943 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.885815 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.947669 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.947618 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:35.966374 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.966343 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:03:35.992130 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:35.992092 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:03:36.056530 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.056490 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume\") pod \"338457c5-d7da-4621-ad5f-efccf2742a75\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " Sep 11 14:03:36.056969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.056551 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq\") pod \"338457c5-d7da-4621-ad5f-efccf2742a75\" (UID: \"338457c5-d7da-4621-ad5f-efccf2742a75\") " Sep 11 14:03:36.056969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.056888 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "338457c5-d7da-4621-ad5f-efccf2742a75" (UID: "338457c5-d7da-4621-ad5f-efccf2742a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:36.058582 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.058551 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq" (OuterVolumeSpecName: "kube-api-access-n8zvq") pod "338457c5-d7da-4621-ad5f-efccf2742a75" (UID: "338457c5-d7da-4621-ad5f-efccf2742a75"). InnerVolumeSpecName "kube-api-access-n8zvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:36.157904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.157864 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qmcw\" (UniqueName: \"kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw\") pod \"195957f5-559e-4b5c-9ada-f9b29fd31e66\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " Sep 11 14:03:36.157904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.157915 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume\") pod \"195957f5-559e-4b5c-9ada-f9b29fd31e66\" (UID: \"195957f5-559e-4b5c-9ada-f9b29fd31e66\") " Sep 11 14:03:36.158169 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.158132 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338457c5-d7da-4621-ad5f-efccf2742a75-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:36.158169 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.158142 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/338457c5-d7da-4621-ad5f-efccf2742a75-kube-api-access-n8zvq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:36.158302 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.158277 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "195957f5-559e-4b5c-9ada-f9b29fd31e66" (UID: "195957f5-559e-4b5c-9ada-f9b29fd31e66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:36.160037 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.160010 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw" (OuterVolumeSpecName: "kube-api-access-4qmcw") pod "195957f5-559e-4b5c-9ada-f9b29fd31e66" (UID: "195957f5-559e-4b5c-9ada-f9b29fd31e66"). InnerVolumeSpecName "kube-api-access-4qmcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:36.222650 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.222516 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:36.258971 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.258927 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qmcw\" (UniqueName: \"kubernetes.io/projected/195957f5-559e-4b5c-9ada-f9b29fd31e66-kube-api-access-4qmcw\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:36.258971 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.258964 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/195957f5-559e-4b5c-9ada-f9b29fd31e66-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:36.423473 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.423436 2519 generic.go:356] "Generic (PLEG): container finished" podID="338457c5-d7da-4621-ad5f-efccf2742a75" containerID="b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15" exitCode=0 Sep 11 14:03:36.423707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.423485 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" Sep 11 14:03:36.423707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.423531 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" event={"ID":"338457c5-d7da-4621-ad5f-efccf2742a75","Type":"ContainerDied","Data":"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15"} Sep 11 14:03:36.423707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.423612 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f" event={"ID":"338457c5-d7da-4621-ad5f-efccf2742a75","Type":"ContainerDied","Data":"81a2931e2d22b09b0ed29d0992abfd75632f80a17effdc0c7395b406dcb582ad"} Sep 11 14:03:36.423707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.423646 2519 scope.go:117] "RemoveContainer" containerID="b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15" Sep 11 14:03:36.424962 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.424944 2519 generic.go:356] "Generic (PLEG): container finished" podID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerID="842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47" exitCode=0 Sep 11 14:03:36.425062 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.424976 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" event={"ID":"195957f5-559e-4b5c-9ada-f9b29fd31e66","Type":"ContainerDied","Data":"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47"} Sep 11 14:03:36.425062 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.424981 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" Sep 11 14:03:36.425062 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.425009 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf" event={"ID":"195957f5-559e-4b5c-9ada-f9b29fd31e66","Type":"ContainerDied","Data":"70fa1513ed7031e024248d95b9c54679947e5b175ef19cfe44401fb2d6dc2daf"} Sep 11 14:03:36.436552 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.436527 2519 scope.go:117] "RemoveContainer" containerID="b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15" Sep 11 14:03:36.436926 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:36.436907 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15\": container with ID starting with b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15 not found: ID does not exist" containerID="b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15" Sep 11 14:03:36.437002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.436934 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15"} err="failed to get container status \"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15\": rpc error: code = NotFound desc = could not find container \"b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15\": container with ID starting with b0f5eca7c6598ca6cd5654a09b5ca1aeae99626ac94f7025a469dbf87f7b7d15 not found: ID does not exist" Sep 11 14:03:36.437002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.436953 2519 scope.go:117] "RemoveContainer" containerID="842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47" Sep 11 14:03:36.445936 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.445916 2519 scope.go:117] "RemoveContainer" containerID="842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47" Sep 11 14:03:36.446230 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:36.446211 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47\": container with ID starting with 842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47 not found: ID does not exist" containerID="842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47" Sep 11 14:03:36.446290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.446237 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47"} err="failed to get container status \"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47\": rpc error: code = NotFound desc = could not find container \"842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47\": container with ID starting with 842c732f427f72118bfb71582bb101caec39cd9c6c0e3a173263bfc462e9ef47 not found: ID does not exist" Sep 11 14:03:36.451453 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.451427 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:03:36.457736 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.457715 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-498d98-566b888b8b-b56jf"] Sep 11 14:03:36.467031 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.466999 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:03:36.472096 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.472073 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-498d98-658857b485-ksp5f"] Sep 11 14:03:36.679857 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.679823 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:36.692172 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.692135 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" path="/var/lib/kubelet/pods/195957f5-559e-4b5c-9ada-f9b29fd31e66/volumes" Sep 11 14:03:36.692559 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:36.692541 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" path="/var/lib/kubelet/pods/338457c5-d7da-4621-ad5f-efccf2742a75/volumes" Sep 11 14:03:37.333480 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.333448 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:37.360550 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.360515 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:37.360826 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.360796 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" podUID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" containerName="payload-processing" containerID="cri-o://ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9" gracePeriod=30 Sep 11 14:03:37.373257 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.373229 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:37.374307 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.374275 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" podUID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" containerName="payload-pre-processing" containerID="cri-o://548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3" gracePeriod=30 Sep 11 14:03:37.407287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.407251 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:37.509242 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.509207 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:37.592395 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.592359 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:37.630981 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.630956 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:37.648566 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.648535 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:37.774421 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.774380 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxh47\" (UniqueName: \"kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47\") pod \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " Sep 11 14:03:37.774627 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.774437 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgb9j\" (UniqueName: \"kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j\") pod \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " Sep 11 14:03:37.774627 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.774600 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume\") pod \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\" (UID: \"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb\") " Sep 11 14:03:37.774737 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.774688 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume\") pod \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\" (UID: \"7ee304fc-dd8f-4f59-ae19-b23fe464a177\") " Sep 11 14:03:37.774910 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.774880 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" (UID: "5da05bff-9a27-40a7-bbe6-7c8754dbd9bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:37.775100 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.775075 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee304fc-dd8f-4f59-ae19-b23fe464a177" (UID: "7ee304fc-dd8f-4f59-ae19-b23fe464a177"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:03:37.775212 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.775172 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:37.775212 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.775191 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee304fc-dd8f-4f59-ae19-b23fe464a177-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:37.776699 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.776669 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j" (OuterVolumeSpecName: "kube-api-access-wgb9j") pod "7ee304fc-dd8f-4f59-ae19-b23fe464a177" (UID: "7ee304fc-dd8f-4f59-ae19-b23fe464a177"). InnerVolumeSpecName "kube-api-access-wgb9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:37.776788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.776718 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47" (OuterVolumeSpecName: "kube-api-access-bxh47") pod "5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" (UID: "5da05bff-9a27-40a7-bbe6-7c8754dbd9bb"). InnerVolumeSpecName "kube-api-access-bxh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:37.876283 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.876187 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh47\" (UniqueName: \"kubernetes.io/projected/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb-kube-api-access-bxh47\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:37.876283 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:37.876222 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgb9j\" (UniqueName: \"kubernetes.io/projected/7ee304fc-dd8f-4f59-ae19-b23fe464a177-kube-api-access-wgb9j\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:38.441814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.441781 2519 generic.go:356] "Generic (PLEG): container finished" podID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" containerID="548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3" exitCode=0 Sep 11 14:03:38.442240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.441829 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" Sep 11 14:03:38.442240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.441865 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" event={"ID":"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb","Type":"ContainerDied","Data":"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3"} Sep 11 14:03:38.442240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.441920 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h" event={"ID":"5da05bff-9a27-40a7-bbe6-7c8754dbd9bb","Type":"ContainerDied","Data":"e6cd33917219ee8f75a800adc8658e1a72fec336344aa36031d2a48884fb835f"} Sep 11 14:03:38.442240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.441955 2519 scope.go:117] "RemoveContainer" containerID="548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3" Sep 11 14:03:38.443219 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.443171 2519 generic.go:356] "Generic (PLEG): container finished" podID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" containerID="ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9" exitCode=0 Sep 11 14:03:38.443309 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.443241 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" Sep 11 14:03:38.443309 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.443269 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" event={"ID":"7ee304fc-dd8f-4f59-ae19-b23fe464a177","Type":"ContainerDied","Data":"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9"} Sep 11 14:03:38.443410 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.443319 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j" event={"ID":"7ee304fc-dd8f-4f59-ae19-b23fe464a177","Type":"ContainerDied","Data":"a1a9db4e0aa8bf96f0d4062465210b3a4fc3b5b1655a66b810284752552fcbfb"} Sep 11 14:03:38.456895 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.456874 2519 scope.go:117] "RemoveContainer" containerID="548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3" Sep 11 14:03:38.457272 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:38.457241 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3\": container with ID starting with 548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3 not found: ID does not exist" containerID="548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3" Sep 11 14:03:38.457352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.457284 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3"} err="failed to get container status \"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3\": rpc error: code = NotFound desc = could not find container \"548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3\": container with ID starting with 548952ae3d7d1a4b5e23dd3f26fa1927ae12b6e731193e0e65fe930ca5737bd3 not found: ID does not exist" Sep 11 14:03:38.457352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.457309 2519 scope.go:117] "RemoveContainer" containerID="ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9" Sep 11 14:03:38.466687 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.466664 2519 scope.go:117] "RemoveContainer" containerID="ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9" Sep 11 14:03:38.467011 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:38.466993 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9\": container with ID starting with ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9 not found: ID does not exist" containerID="ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9" Sep 11 14:03:38.467083 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.467017 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9"} err="failed to get container status \"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9\": rpc error: code = NotFound desc = could not find container \"ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9\": container with ID starting with ac3e7f861f7ea8160627611f0907ea3722ed8a2a24b50f3ca4c7e79a7bd676b9 not found: ID does not exist" Sep 11 14:03:38.467549 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.467530 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:38.472463 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.472437 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bd7f15a9-7468ccdd54-gh49h"] Sep 11 14:03:38.482708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.482681 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:38.484663 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.484643 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bd7f15a9-69fd7f56c7-gzw2j"] Sep 11 14:03:38.656503 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.656470 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:38.691957 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.691879 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" path="/var/lib/kubelet/pods/5da05bff-9a27-40a7-bbe6-7c8754dbd9bb/volumes" Sep 11 14:03:38.692241 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:38.692229 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" path="/var/lib/kubelet/pods/7ee304fc-dd8f-4f59-ae19-b23fe464a177/volumes" Sep 11 14:03:39.075220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:39.075134 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:39.383296 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:39.383263 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:39.637488 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:39.637406 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:39.761123 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:39.761081 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" podUID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" containerName="main" containerID="cri-o://bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e" gracePeriod=45 Sep 11 14:03:40.021414 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.021390 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:40.199344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199252 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199304 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199346 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199647 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199372 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199647 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199456 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwd5l\" (UniqueName: \"kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199647 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199490 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs\") pod \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\" (UID: \"4d8d64a9-0a30-43c2-b614-f3950bd745b4\") " Sep 11 14:03:40.199647 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199506 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home" (OuterVolumeSpecName: "home") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:03:40.199647 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199622 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:03:40.199884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199693 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:03:40.199884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199870 2519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.199983 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199891 2519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.199983 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.199905 2519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.201794 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.201765 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l" (OuterVolumeSpecName: "kube-api-access-bwd5l") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "kube-api-access-bwd5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:03:40.201919 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.201863 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:03:40.201919 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.201894 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm" (OuterVolumeSpecName: "dshm") pod "4d8d64a9-0a30-43c2-b614-f3950bd745b4" (UID: "4d8d64a9-0a30-43c2-b614-f3950bd745b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:03:40.301078 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.301049 2519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d8d64a9-0a30-43c2-b614-f3950bd745b4-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.301078 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.301073 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwd5l\" (UniqueName: \"kubernetes.io/projected/4d8d64a9-0a30-43c2-b614-f3950bd745b4-kube-api-access-bwd5l\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.301296 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.301086 2519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d8d64a9-0a30-43c2-b614-f3950bd745b4-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:03:40.458026 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.457942 2519 generic.go:356] "Generic (PLEG): container finished" podID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" containerID="bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e" exitCode=0 Sep 11 14:03:40.458026 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.457986 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" Sep 11 14:03:40.458203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.458037 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" event={"ID":"4d8d64a9-0a30-43c2-b614-f3950bd745b4","Type":"ContainerDied","Data":"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e"} Sep 11 14:03:40.458203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.458084 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f" event={"ID":"4d8d64a9-0a30-43c2-b614-f3950bd745b4","Type":"ContainerDied","Data":"3f655f4e4a21ad405917513c24c43374050440b7d228eb2514b010d59141b128"} Sep 11 14:03:40.458203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.458107 2519 scope.go:117] "RemoveContainer" containerID="bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e" Sep 11 14:03:40.467242 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.467224 2519 scope.go:117] "RemoveContainer" containerID="bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e" Sep 11 14:03:40.467498 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:03:40.467479 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e\": container with ID starting with bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e not found: ID does not exist" containerID="bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e" Sep 11 14:03:40.467546 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.467508 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e"} err="failed to get container status \"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e\": rpc error: code = NotFound desc = could not find container \"bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e\": container with ID starting with bf197cd7258f542ecde23842ce7ef6b0a463d8edcef9bae3a3e046fa3f5b962e not found: ID does not exist" Sep 11 14:03:40.479104 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.479080 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:03:40.483526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.483506 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-498d98/autores-model-498d98-kserve-6464594d4c-6ss6f"] Sep 11 14:03:40.692725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:40.692692 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" path="/var/lib/kubelet/pods/4d8d64a9-0a30-43c2-b614-f3950bd745b4/volumes" Sep 11 14:03:41.487964 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:41.487933 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:41.589118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:41.589086 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:42.677075 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:42.677043 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:42.774810 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:42.774776 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:42.883753 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:42.883725 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:42.987445 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:42.987361 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:43.175797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.175751 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:43.245460 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.245359 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:43.291206 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291169 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:03:43.291678 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291661 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" containerName="payload-processing" Sep 11 14:03:43.291734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291679 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" containerName="payload-processing" Sep 11 14:03:43.291734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291696 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerName="payload-pre-processing" Sep 11 14:03:43.291734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291702 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerName="payload-pre-processing" Sep 11 14:03:43.291734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291714 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" containerName="payload-processing" Sep 11 14:03:43.291734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291722 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" containerName="payload-processing" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291738 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" containerName="main" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291745 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" containerName="main" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291763 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" containerName="payload-pre-processing" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291771 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" containerName="payload-pre-processing" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291840 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="195957f5-559e-4b5c-9ada-f9b29fd31e66" containerName="payload-pre-processing" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291852 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee304fc-dd8f-4f59-ae19-b23fe464a177" containerName="payload-processing" Sep 11 14:03:43.291887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291888 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da05bff-9a27-40a7-bbe6-7c8754dbd9bb" containerName="payload-pre-processing" Sep 11 14:03:43.292224 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291894 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8d64a9-0a30-43c2-b614-f3950bd745b4" containerName="main" Sep 11 14:03:43.292224 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.291901 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="338457c5-d7da-4621-ad5f-efccf2742a75" containerName="payload-processing" Sep 11 14:03:43.304655 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.304627 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:03:43.304823 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.304758 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.307976 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.307946 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-10d06905\"" Sep 11 14:03:43.308350 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.308314 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.310813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.310792 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:03:43.313507 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.313480 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:03:43.325874 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.325850 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrjq\" (UniqueName: \"kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.325980 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.325943 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.325980 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.325963 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvfn\" (UniqueName: \"kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.326060 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.325987 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.426411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.426380 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.426596 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.426442 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrjq\" (UniqueName: \"kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.426596 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.426497 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.426596 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.426515 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvfn\" (UniqueName: \"kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.427218 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.427193 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.427318 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.427225 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.434665 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.434637 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvfn\" (UniqueName: \"kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn\") pod \"payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.434899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.434884 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrjq\" (UniqueName: \"kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq\") pod \"payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.616846 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.616805 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:43.618726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.618705 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:43.625683 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.625666 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:43.904030 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.903999 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:03:43.905076 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:43.905049 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8f4c9f_4e5d_4356_8b7c_8ce49760b8b3.slice/crio-9acf709ef4e968c651ddb54cddeb42b9e33e931300d77fe430402b837e3a690d WatchSource:0}: Error finding container 9acf709ef4e968c651ddb54cddeb42b9e33e931300d77fe430402b837e3a690d: Status 404 returned error can't find the container with id 9acf709ef4e968c651ddb54cddeb42b9e33e931300d77fe430402b837e3a690d Sep 11 14:03:43.910133 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.910068 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:03:43.911925 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:43.911901 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3612c2_e142_49f9_bbe6_63491dec380f.slice/crio-e2dc83225c73b48962f8e1c8533ad0f6f1ff39e690b0c60a0bd0935ce4f5d6df WatchSource:0}: Error finding container e2dc83225c73b48962f8e1c8533ad0f6f1ff39e690b0c60a0bd0935ce4f5d6df: Status 404 returned error can't find the container with id e2dc83225c73b48962f8e1c8533ad0f6f1ff39e690b0c60a0bd0935ce4f5d6df Sep 11 14:03:43.913394 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:43.913371 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:44.485038 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.485000 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" event={"ID":"1b3612c2-e142-49f9-bbe6-63491dec380f","Type":"ContainerStarted","Data":"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758"} Sep 11 14:03:44.485237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.485065 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" event={"ID":"1b3612c2-e142-49f9-bbe6-63491dec380f","Type":"ContainerStarted","Data":"e2dc83225c73b48962f8e1c8533ad0f6f1ff39e690b0c60a0bd0935ce4f5d6df"} Sep 11 14:03:44.485237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.485101 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:44.486994 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.486966 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" event={"ID":"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3","Type":"ContainerStarted","Data":"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95"} Sep 11 14:03:44.487121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.487015 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" event={"ID":"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3","Type":"ContainerStarted","Data":"9acf709ef4e968c651ddb54cddeb42b9e33e931300d77fe430402b837e3a690d"} Sep 11 14:03:44.487121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.487047 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:03:44.502800 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.502745 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" podStartSLOduration=1.5027253649999999 podStartE2EDuration="1.502725365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:44.501857793 +0000 UTC m=+892.256172684" watchObservedRunningTime="2026-09-11 14:03:44.502725365 +0000 UTC m=+892.257040257" Sep 11 14:03:44.517797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.517731 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" podStartSLOduration=1.517711309 podStartE2EDuration="1.517711309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:44.516807019 +0000 UTC m=+892.271121913" watchObservedRunningTime="2026-09-11 14:03:44.517711309 +0000 UTC m=+892.272026218" Sep 11 14:03:44.559341 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:44.559311 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:47.064743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:47.064615 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:47.164659 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:47.164626 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.233554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.233522 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.335813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.335779 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.561235 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.561109 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.728005 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.727969 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.824428 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.824347 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:48.831222 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.831189 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:03:48.834840 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.834822 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.837244 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.837221 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-93da06-dockercfg-rvh9s\"" Sep 11 14:03:48.837352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.837246 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-93da06\"" Sep 11 14:03:48.844695 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.844545 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:03:48.858926 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.858899 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:03:48.862879 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.862857 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.873936 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.873906 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:03:48.874290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.874269 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdqb\" (UniqueName: \"kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.874399 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.874322 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.874479 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.874439 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.874479 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.874468 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz22\" (UniqueName: \"kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.975190 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975139 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdqb\" (UniqueName: \"kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.975379 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975217 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.975379 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975284 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.975379 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975317 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz22\" (UniqueName: \"kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.976009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975982 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:48.976009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.975996 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.984274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.984244 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdqb\" (UniqueName: \"kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb\") pod \"payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:48.984369 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:48.984302 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz22\" (UniqueName: \"kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22\") pod \"payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:49.148442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.148397 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:49.177630 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.177532 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:49.427568 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.427543 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:03:49.428867 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:49.428844 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d056348_1bc4_4cc3_b436_58e1ed5db2c4.slice/crio-f09293f3d5fc026d93ffb1d23c0c8494666204203ba1889dc9ee0cfdf160ba9a WatchSource:0}: Error finding container f09293f3d5fc026d93ffb1d23c0c8494666204203ba1889dc9ee0cfdf160ba9a: Status 404 returned error can't find the container with id f09293f3d5fc026d93ffb1d23c0c8494666204203ba1889dc9ee0cfdf160ba9a Sep 11 14:03:49.464001 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.463895 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:03:49.468211 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:03:49.468179 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ae410c_7e06_42d2_9c74_f0aa121846fb.slice/crio-90c1cd65eb1a45bce96e24598787a17bf44b67234f94ecd4284afc676bcf036c WatchSource:0}: Error finding container 90c1cd65eb1a45bce96e24598787a17bf44b67234f94ecd4284afc676bcf036c: Status 404 returned error can't find the container with id 90c1cd65eb1a45bce96e24598787a17bf44b67234f94ecd4284afc676bcf036c Sep 11 14:03:49.519824 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.519791 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" event={"ID":"3d056348-1bc4-4cc3-b436-58e1ed5db2c4","Type":"ContainerStarted","Data":"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb"} Sep 11 14:03:49.519970 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.519862 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:03:49.519970 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.519889 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" event={"ID":"3d056348-1bc4-4cc3-b436-58e1ed5db2c4","Type":"ContainerStarted","Data":"f09293f3d5fc026d93ffb1d23c0c8494666204203ba1889dc9ee0cfdf160ba9a"} Sep 11 14:03:49.521185 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.521156 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" event={"ID":"c0ae410c-7e06-42d2-9c74-f0aa121846fb","Type":"ContainerStarted","Data":"90c1cd65eb1a45bce96e24598787a17bf44b67234f94ecd4284afc676bcf036c"} Sep 11 14:03:49.535515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:49.535471 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" podStartSLOduration=1.5354565550000001 podStartE2EDuration="1.535456555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:49.534311857 +0000 UTC m=+897.288626761" watchObservedRunningTime="2026-09-11 14:03:49.535456555 +0000 UTC m=+897.289771445" Sep 11 14:03:50.529409 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:50.529368 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" event={"ID":"c0ae410c-7e06-42d2-9c74-f0aa121846fb","Type":"ContainerStarted","Data":"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9"} Sep 11 14:03:50.530009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:50.529437 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:03:50.546742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:50.546692 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" podStartSLOduration=2.546676094 podStartE2EDuration="2.546676094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:50.544260182 +0000 UTC m=+898.298575074" watchObservedRunningTime="2026-09-11 14:03:50.546676094 +0000 UTC m=+898.300990986" Sep 11 14:03:52.710463 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:52.710429 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:03:52.711535 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:52.711510 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:03:55.495919 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:55.495890 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:03:55.496383 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:03:55.495962 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:04:00.265338 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:00.265299 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:00.532364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:00.532284 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:04:01.538667 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:01.538637 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:04:03.793487 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:03.793455 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:09.130993 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:09.130952 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:10.398479 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:10.398441 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:11.970982 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:11.970949 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:13.046267 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.046231 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:13.699918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.699884 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:13.730957 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.730922 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:04:13.731188 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.731158 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" podUID="1b3612c2-e142-49f9-bbe6-63491dec380f" containerName="payload-processing" containerID="cri-o://4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758" gracePeriod=30 Sep 11 14:04:13.740123 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.740094 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:04:13.740660 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.740420 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" podUID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" containerName="payload-pre-processing" containerID="cri-o://ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95" gracePeriod=30 Sep 11 14:04:13.777915 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.777885 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:13.853566 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.853482 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:13.928525 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:13.928485 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:14.005635 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.005201 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:04:14.007969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.007907 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:14.040132 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.040102 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume\") pod \"1b3612c2-e142-49f9-bbe6-63491dec380f\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " Sep 11 14:04:14.040290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.040200 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grrjq\" (UniqueName: \"kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq\") pod \"1b3612c2-e142-49f9-bbe6-63491dec380f\" (UID: \"1b3612c2-e142-49f9-bbe6-63491dec380f\") " Sep 11 14:04:14.041442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.041414 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b3612c2-e142-49f9-bbe6-63491dec380f" (UID: "1b3612c2-e142-49f9-bbe6-63491dec380f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:04:14.045568 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.045523 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq" (OuterVolumeSpecName: "kube-api-access-grrjq") pod "1b3612c2-e142-49f9-bbe6-63491dec380f" (UID: "1b3612c2-e142-49f9-bbe6-63491dec380f"). InnerVolumeSpecName "kube-api-access-grrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:04:14.050531 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.050507 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:04:14.091081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.090973 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:14.140913 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.140881 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqvfn\" (UniqueName: \"kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn\") pod \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " Sep 11 14:04:14.141068 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.140922 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume\") pod \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\" (UID: \"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3\") " Sep 11 14:04:14.141120 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.141109 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3612c2-e142-49f9-bbe6-63491dec380f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:04:14.141177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.141122 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrjq\" (UniqueName: \"kubernetes.io/projected/1b3612c2-e142-49f9-bbe6-63491dec380f-kube-api-access-grrjq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:04:14.141328 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.141294 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" (UID: "ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:04:14.143071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.143049 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn" (OuterVolumeSpecName: "kube-api-access-pqvfn") pod "ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" (UID: "ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3"). InnerVolumeSpecName "kube-api-access-pqvfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:04:14.241697 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.241604 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqvfn\" (UniqueName: \"kubernetes.io/projected/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-kube-api-access-pqvfn\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:04:14.241697 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.241641 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:04:14.662465 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.662426 2519 generic.go:356] "Generic (PLEG): container finished" podID="1b3612c2-e142-49f9-bbe6-63491dec380f" containerID="4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758" exitCode=0 Sep 11 14:04:14.662677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.662491 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" Sep 11 14:04:14.662677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.662506 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" event={"ID":"1b3612c2-e142-49f9-bbe6-63491dec380f","Type":"ContainerDied","Data":"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758"} Sep 11 14:04:14.662677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.662624 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44" event={"ID":"1b3612c2-e142-49f9-bbe6-63491dec380f","Type":"ContainerDied","Data":"e2dc83225c73b48962f8e1c8533ad0f6f1ff39e690b0c60a0bd0935ce4f5d6df"} Sep 11 14:04:14.662677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.662660 2519 scope.go:117] "RemoveContainer" containerID="4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758" Sep 11 14:04:14.664111 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.664086 2519 generic.go:356] "Generic (PLEG): container finished" podID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" containerID="ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95" exitCode=0 Sep 11 14:04:14.664236 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.664138 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" Sep 11 14:04:14.664236 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.664169 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" event={"ID":"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3","Type":"ContainerDied","Data":"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95"} Sep 11 14:04:14.664236 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.664231 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc" event={"ID":"ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3","Type":"ContainerDied","Data":"9acf709ef4e968c651ddb54cddeb42b9e33e931300d77fe430402b837e3a690d"} Sep 11 14:04:14.674152 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.674082 2519 scope.go:117] "RemoveContainer" containerID="4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758" Sep 11 14:04:14.674449 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:04:14.674413 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758\": container with ID starting with 4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758 not found: ID does not exist" containerID="4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758" Sep 11 14:04:14.674555 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.674458 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758"} err="failed to get container status \"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758\": rpc error: code = NotFound desc = could not find container \"4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758\": container with ID starting with 4f71dc4e27a9bc2c3c4fb6b0419764bf50f6af8ba22e7c5ee8c60bc7e0448758 not found: ID does not exist" Sep 11 14:04:14.674555 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.674481 2519 scope.go:117] "RemoveContainer" containerID="ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95" Sep 11 14:04:14.683240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.683220 2519 scope.go:117] "RemoveContainer" containerID="ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95" Sep 11 14:04:14.683505 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:04:14.683484 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95\": container with ID starting with ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95 not found: ID does not exist" containerID="ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95" Sep 11 14:04:14.683563 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.683510 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95"} err="failed to get container status \"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95\": rpc error: code = NotFound desc = could not find container \"ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95\": container with ID starting with ac51c363dc597f74772425b10d4fb545815d04213d3de0ba2ce689ba86e33b95 not found: ID does not exist" Sep 11 14:04:14.691570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.691549 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:04:14.700082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.700057 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10d06905-5f4bcd7c78-c4jxc"] Sep 11 14:04:14.720105 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.720078 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:04:14.722839 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.722816 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-10d06905-85c56cd6df-c4n44"] Sep 11 14:04:14.747664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.747633 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:14.842554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.842520 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:14.943036 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:14.942955 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:16.260424 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:16.260388 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:16.691821 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:16.691782 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b3612c2-e142-49f9-bbe6-63491dec380f" path="/var/lib/kubelet/pods/1b3612c2-e142-49f9-bbe6-63491dec380f/volumes" Sep 11 14:04:16.692183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:16.692167 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" path="/var/lib/kubelet/pods/ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3/volumes" Sep 11 14:04:22.322063 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:22.322025 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:22.424234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:22.424202 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:22.522770 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:22.522733 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:22.623073 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:22.623037 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:23.020996 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:23.020916 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:33.549396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.549361 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:04:33.549968 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.549950 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b3612c2-e142-49f9-bbe6-63491dec380f" containerName="payload-processing" Sep 11 14:04:33.550013 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.549970 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b3612c2-e142-49f9-bbe6-63491dec380f" containerName="payload-processing" Sep 11 14:04:33.550013 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.549986 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" containerName="payload-pre-processing" Sep 11 14:04:33.550013 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.549991 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" containerName="payload-pre-processing" Sep 11 14:04:33.550112 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.550058 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8f4c9f-4e5d-4356-8b7c-8ce49760b8b3" containerName="payload-pre-processing" Sep 11 14:04:33.550112 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.550068 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b3612c2-e142-49f9-bbe6-63491dec380f" containerName="payload-processing" Sep 11 14:04:33.553756 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.553736 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.555928 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.555906 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-93da06\"/\"openshift-service-ca.crt\"" Sep 11 14:04:33.556045 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.555944 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-93da06\"/\"kube-root-ca.crt\"" Sep 11 14:04:33.556737 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.556716 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-93da06\"/\"default-dockercfg-ln9gf\"" Sep 11 14:04:33.556865 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.556816 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-93da06\"/\"explicit-model-93da06-kserve-self-signed-certs\"" Sep 11 14:04:33.561695 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.561666 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:04:33.620086 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620056 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9msd\" (UniqueName: \"kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.620274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620099 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.620274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620120 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.620274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620201 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.620274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620260 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.620485 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.620343 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721057 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721021 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721229 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721062 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721229 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721102 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721229 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721168 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721239 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9msd\" (UniqueName: \"kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721286 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721564 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721536 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721551 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.721895 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.721874 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.723688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.723659 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.723803 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.723695 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.729174 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.729151 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9msd\" (UniqueName: \"kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd\") pod \"explicit-model-93da06-kserve-6c599d667f-2jxhh\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:33.755501 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.755481 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:33.865083 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:33.865051 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:34.138150 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:34.138123 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:04:34.140097 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:04:34.140067 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76a2410a_a605_4b50_8595_aedf403a7d1c.slice/crio-c956c25b28f9035fc9562134b21c8153cab78daf7b0fed17cf132a7527933a06 WatchSource:0}: Error finding container c956c25b28f9035fc9562134b21c8153cab78daf7b0fed17cf132a7527933a06: Status 404 returned error can't find the container with id c956c25b28f9035fc9562134b21c8153cab78daf7b0fed17cf132a7527933a06 Sep 11 14:04:34.771209 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:34.771173 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" event={"ID":"76a2410a-a605-4b50-8595-aedf403a7d1c","Type":"ContainerStarted","Data":"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e"} Sep 11 14:04:34.771615 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:34.771220 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" event={"ID":"76a2410a-a605-4b50-8595-aedf403a7d1c","Type":"ContainerStarted","Data":"c956c25b28f9035fc9562134b21c8153cab78daf7b0fed17cf132a7527933a06"} Sep 11 14:04:34.790846 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:34.790799 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" podStartSLOduration=1.790783286 podStartE2EDuration="1.790783286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:04:34.787931721 +0000 UTC m=+942.542246612" watchObservedRunningTime="2026-09-11 14:04:34.790783286 +0000 UTC m=+942.545098176" Sep 11 14:04:43.865454 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:43.865405 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:43.865454 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:43.865467 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:43.878568 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:43.878543 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:44.837768 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:44.837738 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:04:48.947527 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:48.947494 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:54.742648 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:54.742597 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:54.843461 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:54.843421 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:54.967369 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:04:54.967331 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:07.548518 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:07.548484 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:05:07.573444 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:07.573415 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:07.792547 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:07.792518 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:10.771068 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:10.771029 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:16.474830 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:16.474759 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:16.540765 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:16.540735 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:16.627509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:16.627482 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:16.725224 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:16.725140 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:18.198622 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.198552 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:18.214383 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.214336 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:05:18.214672 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.214644 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" podUID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" containerName="payload-processing" containerID="cri-o://dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9" gracePeriod=30 Sep 11 14:05:18.223528 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.222497 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:05:18.223528 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.222758 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" podUID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" containerName="payload-pre-processing" containerID="cri-o://d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb" gracePeriod=30 Sep 11 14:05:18.300921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.300792 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:18.398687 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.398619 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:18.457941 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.457855 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:18.498923 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.498902 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:05:18.501842 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.501822 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:05:18.556079 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556052 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmz22\" (UniqueName: \"kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22\") pod \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " Sep 11 14:05:18.556253 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556103 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume\") pod \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\" (UID: \"3d056348-1bc4-4cc3-b436-58e1ed5db2c4\") " Sep 11 14:05:18.556253 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556128 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume\") pod \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " Sep 11 14:05:18.556253 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556231 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzdqb\" (UniqueName: \"kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb\") pod \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\" (UID: \"c0ae410c-7e06-42d2-9c74-f0aa121846fb\") " Sep 11 14:05:18.556536 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556494 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d056348-1bc4-4cc3-b436-58e1ed5db2c4" (UID: "3d056348-1bc4-4cc3-b436-58e1ed5db2c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:18.556691 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.556555 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0ae410c-7e06-42d2-9c74-f0aa121846fb" (UID: "c0ae410c-7e06-42d2-9c74-f0aa121846fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:18.558250 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.558231 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22" (OuterVolumeSpecName: "kube-api-access-bmz22") pod "3d056348-1bc4-4cc3-b436-58e1ed5db2c4" (UID: "3d056348-1bc4-4cc3-b436-58e1ed5db2c4"). InnerVolumeSpecName "kube-api-access-bmz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:18.558339 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.558313 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb" (OuterVolumeSpecName: "kube-api-access-wzdqb") pod "c0ae410c-7e06-42d2-9c74-f0aa121846fb" (UID: "c0ae410c-7e06-42d2-9c74-f0aa121846fb"). InnerVolumeSpecName "kube-api-access-wzdqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:18.657921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.657887 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdqb\" (UniqueName: \"kubernetes.io/projected/c0ae410c-7e06-42d2-9c74-f0aa121846fb-kube-api-access-wzdqb\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:18.657921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.657915 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz22\" (UniqueName: \"kubernetes.io/projected/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-kube-api-access-bmz22\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:18.657921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.657928 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d056348-1bc4-4cc3-b436-58e1ed5db2c4-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:18.658148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.657943 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ae410c-7e06-42d2-9c74-f0aa121846fb-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:18.747460 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:18.747391 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:19.011081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.010995 2519 generic.go:356] "Generic (PLEG): container finished" podID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" containerID="dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9" exitCode=0 Sep 11 14:05:19.011081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.011035 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" Sep 11 14:05:19.011281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.011086 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" event={"ID":"c0ae410c-7e06-42d2-9c74-f0aa121846fb","Type":"ContainerDied","Data":"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9"} Sep 11 14:05:19.011281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.011131 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7" event={"ID":"c0ae410c-7e06-42d2-9c74-f0aa121846fb","Type":"ContainerDied","Data":"90c1cd65eb1a45bce96e24598787a17bf44b67234f94ecd4284afc676bcf036c"} Sep 11 14:05:19.011281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.011153 2519 scope.go:117] "RemoveContainer" containerID="dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9" Sep 11 14:05:19.012515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.012491 2519 generic.go:356] "Generic (PLEG): container finished" podID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" containerID="d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb" exitCode=0 Sep 11 14:05:19.012649 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.012528 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" Sep 11 14:05:19.012649 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.012549 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" event={"ID":"3d056348-1bc4-4cc3-b436-58e1ed5db2c4","Type":"ContainerDied","Data":"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb"} Sep 11 14:05:19.012649 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.012609 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22" event={"ID":"3d056348-1bc4-4cc3-b436-58e1ed5db2c4","Type":"ContainerDied","Data":"f09293f3d5fc026d93ffb1d23c0c8494666204203ba1889dc9ee0cfdf160ba9a"} Sep 11 14:05:19.020378 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.020361 2519 scope.go:117] "RemoveContainer" containerID="dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9" Sep 11 14:05:19.020658 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:19.020637 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9\": container with ID starting with dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9 not found: ID does not exist" containerID="dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9" Sep 11 14:05:19.020708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.020665 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9"} err="failed to get container status \"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9\": rpc error: code = NotFound desc = could not find container \"dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9\": container with ID starting with dcc062224dbc13075b97ab58a63e6176d8e570ca7bfc895793f1840d7450bba9 not found: ID does not exist" Sep 11 14:05:19.020708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.020680 2519 scope.go:117] "RemoveContainer" containerID="d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb" Sep 11 14:05:19.028303 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.028282 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:05:19.029247 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.029235 2519 scope.go:117] "RemoveContainer" containerID="d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb" Sep 11 14:05:19.029524 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:19.029506 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb\": container with ID starting with d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb not found: ID does not exist" containerID="d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb" Sep 11 14:05:19.029602 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.029533 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb"} err="failed to get container status \"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb\": rpc error: code = NotFound desc = could not find container \"d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb\": container with ID starting with d8bcf2c6c6bb81fb38563de1479c41d2f7c3417f37771231b91da9c9ce32f1fb not found: ID does not exist" Sep 11 14:05:19.034072 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.034052 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-93da06-f9c679f9d-47x22"] Sep 11 14:05:19.043023 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.042975 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:05:19.044931 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.044912 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-93da06-5b799f5fc7-vfxg7"] Sep 11 14:05:19.791708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:19.791677 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:20.696332 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.696280 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" path="/var/lib/kubelet/pods/3d056348-1bc4-4cc3-b436-58e1ed5db2c4/volumes" Sep 11 14:05:20.696875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.696855 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" path="/var/lib/kubelet/pods/c0ae410c-7e06-42d2-9c74-f0aa121846fb/volumes" Sep 11 14:05:20.722009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.721977 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:20.860024 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.859987 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9"] Sep 11 14:05:20.860675 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860653 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" containerName="payload-pre-processing" Sep 11 14:05:20.860675 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860675 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" containerName="payload-pre-processing" Sep 11 14:05:20.860850 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860705 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" containerName="payload-processing" Sep 11 14:05:20.860850 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860713 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" containerName="payload-processing" Sep 11 14:05:20.860850 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860827 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d056348-1bc4-4cc3-b436-58e1ed5db2c4" containerName="payload-pre-processing" Sep 11 14:05:20.860850 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.860843 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ae410c-7e06-42d2-9c74-f0aa121846fb" containerName="payload-processing" Sep 11 14:05:20.864264 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.864241 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:20.868561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.868529 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:05:20.882926 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.882888 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9"] Sep 11 14:05:20.978866 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:20.978777 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9\" (UID: \"63449b6a-e0b1-4f10-b20c-881904a81b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:21.068856 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:21.068819 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:21.080152 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:21.080116 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9\" (UID: \"63449b6a-e0b1-4f10-b20c-881904a81b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:21.080822 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:21.080799 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9\" (UID: \"63449b6a-e0b1-4f10-b20c-881904a81b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:21.175206 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:21.175168 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:21.446958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:21.446932 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9"] Sep 11 14:05:21.449617 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:05:21.449589 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63449b6a_e0b1_4f10_b20c_881904a81b51.slice/crio-2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66 WatchSource:0}: Error finding container 2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66: Status 404 returned error can't find the container with id 2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66 Sep 11 14:05:22.032127 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.032030 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" event={"ID":"63449b6a-e0b1-4f10-b20c-881904a81b51","Type":"ContainerStarted","Data":"14493c87102823888f64fbacac453af6a2417b9ac008a375366c92921b13a2b4"} Sep 11 14:05:22.032127 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.032084 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" event={"ID":"63449b6a-e0b1-4f10-b20c-881904a81b51","Type":"ContainerStarted","Data":"2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66"} Sep 11 14:05:22.047017 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.046965 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" podStartSLOduration=2.0469466 podStartE2EDuration="2.0469466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:22.045877339 +0000 UTC m=+989.800192231" watchObservedRunningTime="2026-09-11 14:05:22.0469466 +0000 UTC m=+989.801261718" Sep 11 14:05:22.568107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.568071 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" podUID="76a2410a-a605-4b50-8595-aedf403a7d1c" containerName="main" containerID="cri-o://a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e" gracePeriod=45 Sep 11 14:05:22.822064 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.822007 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:05:22.897590 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897548 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.897789 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897621 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.897789 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897665 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.897789 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897736 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.897789 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897780 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9msd\" (UniqueName: \"kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.898002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897797 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm\") pod \"76a2410a-a605-4b50-8595-aedf403a7d1c\" (UID: \"76a2410a-a605-4b50-8595-aedf403a7d1c\") " Sep 11 14:05:22.898002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897881 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:22.898002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897915 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:22.898002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.897923 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home" (OuterVolumeSpecName: "home") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:22.898140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.898094 2519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:22.898140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.898116 2519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:22.898140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.898129 2519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:22.900034 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.900004 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd" (OuterVolumeSpecName: "kube-api-access-k9msd") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "kube-api-access-k9msd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:22.900131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.900039 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm" (OuterVolumeSpecName: "dshm") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:22.900131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.900075 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76a2410a-a605-4b50-8595-aedf403a7d1c" (UID: "76a2410a-a605-4b50-8595-aedf403a7d1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:05:22.998920 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.998878 2519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a2410a-a605-4b50-8595-aedf403a7d1c-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:22.998920 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.998913 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9msd\" (UniqueName: \"kubernetes.io/projected/76a2410a-a605-4b50-8595-aedf403a7d1c-kube-api-access-k9msd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:22.998920 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:22.998929 2519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a2410a-a605-4b50-8595-aedf403a7d1c-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:23.038800 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.038757 2519 generic.go:356] "Generic (PLEG): container finished" podID="76a2410a-a605-4b50-8595-aedf403a7d1c" containerID="a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e" exitCode=0 Sep 11 14:05:23.038800 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.038802 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" Sep 11 14:05:23.039320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.038823 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" event={"ID":"76a2410a-a605-4b50-8595-aedf403a7d1c","Type":"ContainerDied","Data":"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e"} Sep 11 14:05:23.039320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.038866 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh" event={"ID":"76a2410a-a605-4b50-8595-aedf403a7d1c","Type":"ContainerDied","Data":"c956c25b28f9035fc9562134b21c8153cab78daf7b0fed17cf132a7527933a06"} Sep 11 14:05:23.039320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.038888 2519 scope.go:117] "RemoveContainer" containerID="a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e" Sep 11 14:05:23.040903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.040875 2519 generic.go:356] "Generic (PLEG): container finished" podID="63449b6a-e0b1-4f10-b20c-881904a81b51" containerID="14493c87102823888f64fbacac453af6a2417b9ac008a375366c92921b13a2b4" exitCode=0 Sep 11 14:05:23.041057 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.041026 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" event={"ID":"63449b6a-e0b1-4f10-b20c-881904a81b51","Type":"ContainerDied","Data":"14493c87102823888f64fbacac453af6a2417b9ac008a375366c92921b13a2b4"} Sep 11 14:05:23.048831 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.048812 2519 scope.go:117] "RemoveContainer" containerID="a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e" Sep 11 14:05:23.049113 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:23.049096 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e\": container with ID starting with a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e not found: ID does not exist" containerID="a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e" Sep 11 14:05:23.049160 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.049121 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e"} err="failed to get container status \"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e\": rpc error: code = NotFound desc = could not find container \"a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e\": container with ID starting with a965db14152fe34697733521a2e40561f599d58208303c6f5df1bc63487fb09e not found: ID does not exist" Sep 11 14:05:23.070451 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.070406 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:05:23.077442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:23.077363 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-93da06/explicit-model-93da06-kserve-6c599d667f-2jxhh"] Sep 11 14:05:24.182384 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:24.182359 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:24.312487 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:24.312408 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca\") pod \"63449b6a-e0b1-4f10-b20c-881904a81b51\" (UID: \"63449b6a-e0b1-4f10-b20c-881904a81b51\") " Sep 11 14:05:24.312784 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:24.312760 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63449b6a-e0b1-4f10-b20c-881904a81b51" (UID: "63449b6a-e0b1-4f10-b20c-881904a81b51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:24.413491 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:24.413460 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63449b6a-e0b1-4f10-b20c-881904a81b51-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:24.691996 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:24.691964 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a2410a-a605-4b50-8595-aedf403a7d1c" path="/var/lib/kubelet/pods/76a2410a-a605-4b50-8595-aedf403a7d1c/volumes" Sep 11 14:05:25.035988 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.035902 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:25.052797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.052772 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" Sep 11 14:05:25.052941 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.052812 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9" event={"ID":"63449b6a-e0b1-4f10-b20c-881904a81b51","Type":"ContainerDied","Data":"2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66"} Sep 11 14:05:25.052941 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.052857 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5db79771c8517e798ceadccedbc422b4ffc5aa79173b830cc83775d2497f66" Sep 11 14:05:25.137244 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.137212 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:25.202751 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.202722 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:25.587483 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.587450 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:25.905062 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:25.904985 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:26.169950 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:26.169871 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:27.017774 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:27.017741 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:27.114314 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:27.114286 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:27.215117 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:27.215084 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:27.319708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:27.319616 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:27.937408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:27.937372 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.447287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.447255 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.483408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.483372 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:05:31.483656 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.483629 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" podUID="975dd800-8d43-48ee-85ec-f2ce324d8086" containerName="payload-processing" containerID="cri-o://b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6" gracePeriod=30 Sep 11 14:05:31.489641 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.489610 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:05:31.489878 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.489846 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" podUID="c64ab1f4-c994-421f-8b6f-1fb204982703" containerName="payload-pre-processing" containerID="cri-o://f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556" gracePeriod=30 Sep 11 14:05:31.522433 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.522401 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.595313 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.595274 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.692328 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.691643 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9"] Sep 11 14:05:31.699085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.698909 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.701246 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.701222 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c9fc1-622j9"] Sep 11 14:05:31.772102 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.772074 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:31.774428 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.774400 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:05:31.777236 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.777214 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:05:31.886813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.886785 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume\") pod \"c64ab1f4-c994-421f-8b6f-1fb204982703\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " Sep 11 14:05:31.886984 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.886849 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcn4g\" (UniqueName: \"kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g\") pod \"975dd800-8d43-48ee-85ec-f2ce324d8086\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " Sep 11 14:05:31.886984 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.886914 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume\") pod \"975dd800-8d43-48ee-85ec-f2ce324d8086\" (UID: \"975dd800-8d43-48ee-85ec-f2ce324d8086\") " Sep 11 14:05:31.887066 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.886989 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xswk4\" (UniqueName: \"kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4\") pod \"c64ab1f4-c994-421f-8b6f-1fb204982703\" (UID: \"c64ab1f4-c994-421f-8b6f-1fb204982703\") " Sep 11 14:05:31.887292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.887256 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c64ab1f4-c994-421f-8b6f-1fb204982703" (UID: "c64ab1f4-c994-421f-8b6f-1fb204982703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:31.887409 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.887333 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "975dd800-8d43-48ee-85ec-f2ce324d8086" (UID: "975dd800-8d43-48ee-85ec-f2ce324d8086"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:31.888986 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.888959 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g" (OuterVolumeSpecName: "kube-api-access-lcn4g") pod "975dd800-8d43-48ee-85ec-f2ce324d8086" (UID: "975dd800-8d43-48ee-85ec-f2ce324d8086"). InnerVolumeSpecName "kube-api-access-lcn4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:31.889093 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.889016 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4" (OuterVolumeSpecName: "kube-api-access-xswk4") pod "c64ab1f4-c994-421f-8b6f-1fb204982703" (UID: "c64ab1f4-c994-421f-8b6f-1fb204982703"). InnerVolumeSpecName "kube-api-access-xswk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:31.988613 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.988510 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcn4g\" (UniqueName: \"kubernetes.io/projected/975dd800-8d43-48ee-85ec-f2ce324d8086-kube-api-access-lcn4g\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:31.988613 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.988536 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975dd800-8d43-48ee-85ec-f2ce324d8086-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:31.988613 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.988547 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswk4\" (UniqueName: \"kubernetes.io/projected/c64ab1f4-c994-421f-8b6f-1fb204982703-kube-api-access-xswk4\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:31.988613 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:31.988556 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64ab1f4-c994-421f-8b6f-1fb204982703-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:32.034945 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.034916 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:32.091806 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.091773 2519 generic.go:356] "Generic (PLEG): container finished" podID="c64ab1f4-c994-421f-8b6f-1fb204982703" containerID="f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556" exitCode=0 Sep 11 14:05:32.091958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.091821 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" Sep 11 14:05:32.091958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.091856 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" event={"ID":"c64ab1f4-c994-421f-8b6f-1fb204982703","Type":"ContainerDied","Data":"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556"} Sep 11 14:05:32.091958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.091932 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx" event={"ID":"c64ab1f4-c994-421f-8b6f-1fb204982703","Type":"ContainerDied","Data":"bd7801d07b40a5c38e2733da5feee209309074c844c1b7431fc98d2283af7e1a"} Sep 11 14:05:32.091958 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.091955 2519 scope.go:117] "RemoveContainer" containerID="f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556" Sep 11 14:05:32.093299 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.093273 2519 generic.go:356] "Generic (PLEG): container finished" podID="975dd800-8d43-48ee-85ec-f2ce324d8086" containerID="b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6" exitCode=0 Sep 11 14:05:32.093438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.093301 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" event={"ID":"975dd800-8d43-48ee-85ec-f2ce324d8086","Type":"ContainerDied","Data":"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6"} Sep 11 14:05:32.093438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.093321 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" Sep 11 14:05:32.093438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.093344 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2" event={"ID":"975dd800-8d43-48ee-85ec-f2ce324d8086","Type":"ContainerDied","Data":"80d8be9d2bc296167a0a459ddc71e99c7f51706d005e63947407a12e37136542"} Sep 11 14:05:32.103116 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.103018 2519 scope.go:117] "RemoveContainer" containerID="f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556" Sep 11 14:05:32.103434 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:32.103412 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556\": container with ID starting with f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556 not found: ID does not exist" containerID="f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556" Sep 11 14:05:32.103510 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.103441 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556"} err="failed to get container status \"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556\": rpc error: code = NotFound desc = could not find container \"f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556\": container with ID starting with f8ffecd8f28ba283d6166ed9ff3fd7f484d84707d19fa69e557460129e2dd556 not found: ID does not exist" Sep 11 14:05:32.103510 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.103456 2519 scope.go:117] "RemoveContainer" containerID="b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6" Sep 11 14:05:32.111650 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.111630 2519 scope.go:117] "RemoveContainer" containerID="b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6" Sep 11 14:05:32.111925 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:32.111907 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6\": container with ID starting with b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6 not found: ID does not exist" containerID="b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6" Sep 11 14:05:32.111994 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.111932 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6"} err="failed to get container status \"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6\": rpc error: code = NotFound desc = could not find container \"b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6\": container with ID starting with b81e3a34c453aac59c289cc9bb113ec7e2048c626be8ef8d056af31d9963ada6 not found: ID does not exist" Sep 11 14:05:32.113383 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.113361 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:05:32.120321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.120300 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c9fc1-555c77f6db-9xfvx"] Sep 11 14:05:32.129410 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.129390 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:05:32.134235 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.134214 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c9fc1-6fcd5bbc47-5kwn2"] Sep 11 14:05:32.692273 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.692240 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63449b6a-e0b1-4f10-b20c-881904a81b51" path="/var/lib/kubelet/pods/63449b6a-e0b1-4f10-b20c-881904a81b51/volumes" Sep 11 14:05:32.692673 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.692626 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975dd800-8d43-48ee-85ec-f2ce324d8086" path="/var/lib/kubelet/pods/975dd800-8d43-48ee-85ec-f2ce324d8086/volumes" Sep 11 14:05:32.692947 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:32.692934 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64ab1f4-c994-421f-8b6f-1fb204982703" path="/var/lib/kubelet/pods/c64ab1f4-c994-421f-8b6f-1fb204982703/volumes" Sep 11 14:05:33.333658 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:33.333627 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:33.961855 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:33.961820 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:34.067846 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:34.067804 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:34.200080 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:34.200051 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:34.318752 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:34.318666 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:34.928051 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:34.928023 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:35.025188 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:35.025153 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:35.125637 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:35.125602 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:36.620195 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:36.620158 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:37.261828 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.261778 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:37.262465 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262442 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975dd800-8d43-48ee-85ec-f2ce324d8086" containerName="payload-processing" Sep 11 14:05:37.262465 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262466 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="975dd800-8d43-48ee-85ec-f2ce324d8086" containerName="payload-processing" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262495 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64ab1f4-c994-421f-8b6f-1fb204982703" containerName="payload-pre-processing" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262505 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64ab1f4-c994-421f-8b6f-1fb204982703" containerName="payload-pre-processing" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262520 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a2410a-a605-4b50-8595-aedf403a7d1c" containerName="main" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262528 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a2410a-a605-4b50-8595-aedf403a7d1c" containerName="main" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262538 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63449b6a-e0b1-4f10-b20c-881904a81b51" containerName="revoke-keys" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262546 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="63449b6a-e0b1-4f10-b20c-881904a81b51" containerName="revoke-keys" Sep 11 14:05:37.262664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262654 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a2410a-a605-4b50-8595-aedf403a7d1c" containerName="main" Sep 11 14:05:37.262904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262669 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="63449b6a-e0b1-4f10-b20c-881904a81b51" containerName="revoke-keys" Sep 11 14:05:37.262904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262679 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="975dd800-8d43-48ee-85ec-f2ce324d8086" containerName="payload-processing" Sep 11 14:05:37.262904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.262692 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64ab1f4-c994-421f-8b6f-1fb204982703" containerName="payload-pre-processing" Sep 11 14:05:37.267582 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.267549 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.270732 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.270708 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4edac4\"/\"noten-model-4edac4-kserve-self-signed-certs\"" Sep 11 14:05:37.271003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.270747 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4edac4\"/\"kube-root-ca.crt\"" Sep 11 14:05:37.271118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.270755 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4edac4\"/\"openshift-service-ca.crt\"" Sep 11 14:05:37.271176 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.270820 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4edac4\"/\"default-dockercfg-gvssz\"" Sep 11 14:05:37.276725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.276704 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:37.300351 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.300323 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:37.439898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.439863 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.439898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.439900 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.440114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.439917 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.440114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.440010 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.440114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.440086 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.440114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.440112 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hcz\" (UniqueName: \"kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541178 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541067 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541178 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541117 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541447 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541415 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541529 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541506 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541523 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541684 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541639 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541685 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hcz\" (UniqueName: \"kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.541743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.541688 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.542036 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.542012 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.543967 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.543940 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.544279 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.544259 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.549152 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.549128 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hcz\" (UniqueName: \"kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz\") pod \"noten-model-4edac4-kserve-84f5ff584c-fftn5\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.578337 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.578313 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:37.849269 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:37.849240 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:37.851308 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:05:37.851285 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod576e5173_f39a_44f1_924f_ffff8d9ae836.slice/crio-02bfe134e44226b667dd253ba4662cd92d12acb01157dc4426612c06cf52d92e WatchSource:0}: Error finding container 02bfe134e44226b667dd253ba4662cd92d12acb01157dc4426612c06cf52d92e: Status 404 returned error can't find the container with id 02bfe134e44226b667dd253ba4662cd92d12acb01157dc4426612c06cf52d92e Sep 11 14:05:38.131997 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.131905 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" event={"ID":"576e5173-f39a-44f1-924f-ffff8d9ae836","Type":"ContainerStarted","Data":"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa"} Sep 11 14:05:38.131997 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.131954 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" event={"ID":"576e5173-f39a-44f1-924f-ffff8d9ae836","Type":"ContainerStarted","Data":"02bfe134e44226b667dd253ba4662cd92d12acb01157dc4426612c06cf52d92e"} Sep 11 14:05:38.148178 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.148128 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" podStartSLOduration=1.148108485 podStartE2EDuration="1.148108485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:38.146891889 +0000 UTC m=+1005.901206782" watchObservedRunningTime="2026-09-11 14:05:38.148108485 +0000 UTC m=+1005.902423377" Sep 11 14:05:38.405145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.405065 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:38.505886 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.505854 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:38.763432 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.763348 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:38.861863 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:38.861821 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:39.038156 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.038060 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh"] Sep 11 14:05:39.041836 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.041812 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:39.043863 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.043844 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:05:39.050154 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.049930 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh"] Sep 11 14:05:39.055652 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.055630 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh\" (UID: \"f47b04c8-3ba9-400a-89a2-b7208a3d4763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:39.156380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.156340 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh\" (UID: \"f47b04c8-3ba9-400a-89a2-b7208a3d4763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:39.156995 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.156975 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh\" (UID: \"f47b04c8-3ba9-400a-89a2-b7208a3d4763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:39.165924 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.165899 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:39.295703 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.295628 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:39.318753 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.318718 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:39.354282 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.354254 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:39.446167 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.446134 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:39.632882 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.632856 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh"] Sep 11 14:05:39.634919 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:05:39.634888 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47b04c8_3ba9_400a_89a2_b7208a3d4763.slice/crio-1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18 WatchSource:0}: Error finding container 1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18: Status 404 returned error can't find the container with id 1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18 Sep 11 14:05:39.664925 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.664898 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:39.781716 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:39.781683 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:40.144803 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:40.144763 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" event={"ID":"f47b04c8-3ba9-400a-89a2-b7208a3d4763","Type":"ContainerStarted","Data":"b47c82e345e2278b4558e073b357d93e3ae2c2b72ac9babefef64ff72c40adbe"} Sep 11 14:05:40.145365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:40.144829 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" event={"ID":"f47b04c8-3ba9-400a-89a2-b7208a3d4763","Type":"ContainerStarted","Data":"1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18"} Sep 11 14:05:40.159805 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:40.159759 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" podStartSLOduration=1.159742095 podStartE2EDuration="1.159742095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:40.158354839 +0000 UTC m=+1007.912669731" watchObservedRunningTime="2026-09-11 14:05:40.159742095 +0000 UTC m=+1007.914056990" Sep 11 14:05:41.158554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:41.158511 2519 generic.go:356] "Generic (PLEG): container finished" podID="f47b04c8-3ba9-400a-89a2-b7208a3d4763" containerID="b47c82e345e2278b4558e073b357d93e3ae2c2b72ac9babefef64ff72c40adbe" exitCode=0 Sep 11 14:05:41.158554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:41.158556 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" event={"ID":"f47b04c8-3ba9-400a-89a2-b7208a3d4763","Type":"ContainerDied","Data":"b47c82e345e2278b4558e073b357d93e3ae2c2b72ac9babefef64ff72c40adbe"} Sep 11 14:05:42.295176 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.295150 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:42.380657 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.380627 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca\") pod \"f47b04c8-3ba9-400a-89a2-b7208a3d4763\" (UID: \"f47b04c8-3ba9-400a-89a2-b7208a3d4763\") " Sep 11 14:05:42.380960 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.380937 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f47b04c8-3ba9-400a-89a2-b7208a3d4763" (UID: "f47b04c8-3ba9-400a-89a2-b7208a3d4763"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:42.401738 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.401704 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:42.481509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.481428 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47b04c8-3ba9-400a-89a2-b7208a3d4763-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:42.501875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:42.501842 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:43.077765 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:43.077733 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:43.171146 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:43.171118 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" Sep 11 14:05:43.171327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:43.171164 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh" event={"ID":"f47b04c8-3ba9-400a-89a2-b7208a3d4763","Type":"ContainerDied","Data":"1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18"} Sep 11 14:05:43.171327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:43.171212 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1a7cbf30bfa86ac7217b0c5198c318b38554823c8247275215f624c8dd4b18" Sep 11 14:05:44.837967 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:44.837914 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:47.579424 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:47.579382 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:48.924277 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:48.924240 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:48.987755 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:48.987724 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.169355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.169317 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.465527 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.465491 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.632743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.632712 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.659524 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.659493 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:05:49.659791 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.659760 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" podUID="2282259a-a663-4e86-b50f-80edfc7e5c03" containerName="payload-processing" containerID="cri-o://05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75" gracePeriod=30 Sep 11 14:05:49.669809 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.669784 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:05:49.670028 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.670005 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" podUID="21c6b2e7-e755-4817-9284-606df07c2887" containerName="payload-pre-processing" containerID="cri-o://d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934" gracePeriod=30 Sep 11 14:05:49.706445 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.706373 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.772651 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.772622 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.849500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.849466 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh"] Sep 11 14:05:49.852853 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.852826 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.853806 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.853783 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-da9534-rpzgh"] Sep 11 14:05:49.916031 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.915997 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:49.947689 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.947664 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:05:49.950602 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:49.950556 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:05:50.051238 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051167 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zksl\" (UniqueName: \"kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl\") pod \"2282259a-a663-4e86-b50f-80edfc7e5c03\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " Sep 11 14:05:50.051389 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051264 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume\") pod \"2282259a-a663-4e86-b50f-80edfc7e5c03\" (UID: \"2282259a-a663-4e86-b50f-80edfc7e5c03\") " Sep 11 14:05:50.051389 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051292 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume\") pod \"21c6b2e7-e755-4817-9284-606df07c2887\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " Sep 11 14:05:50.051389 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051343 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2bw7\" (UniqueName: \"kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7\") pod \"21c6b2e7-e755-4817-9284-606df07c2887\" (UID: \"21c6b2e7-e755-4817-9284-606df07c2887\") " Sep 11 14:05:50.051659 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051634 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2282259a-a663-4e86-b50f-80edfc7e5c03" (UID: "2282259a-a663-4e86-b50f-80edfc7e5c03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:50.051746 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.051712 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21c6b2e7-e755-4817-9284-606df07c2887" (UID: "21c6b2e7-e755-4817-9284-606df07c2887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:50.053135 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.053116 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl" (OuterVolumeSpecName: "kube-api-access-9zksl") pod "2282259a-a663-4e86-b50f-80edfc7e5c03" (UID: "2282259a-a663-4e86-b50f-80edfc7e5c03"). InnerVolumeSpecName "kube-api-access-9zksl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:50.053350 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.053327 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7" (OuterVolumeSpecName: "kube-api-access-g2bw7") pod "21c6b2e7-e755-4817-9284-606df07c2887" (UID: "21c6b2e7-e755-4817-9284-606df07c2887"). InnerVolumeSpecName "kube-api-access-g2bw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:50.153003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.152969 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zksl\" (UniqueName: \"kubernetes.io/projected/2282259a-a663-4e86-b50f-80edfc7e5c03-kube-api-access-9zksl\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.153003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.152996 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2282259a-a663-4e86-b50f-80edfc7e5c03-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.153003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.153006 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c6b2e7-e755-4817-9284-606df07c2887-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.153003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.153014 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bw7\" (UniqueName: \"kubernetes.io/projected/21c6b2e7-e755-4817-9284-606df07c2887-kube-api-access-g2bw7\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.207551 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.207514 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:50.208710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.208685 2519 generic.go:356] "Generic (PLEG): container finished" podID="21c6b2e7-e755-4817-9284-606df07c2887" containerID="d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934" exitCode=0 Sep 11 14:05:50.208841 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.208729 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" Sep 11 14:05:50.208841 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.208766 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" event={"ID":"21c6b2e7-e755-4817-9284-606df07c2887","Type":"ContainerDied","Data":"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934"} Sep 11 14:05:50.208841 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.208815 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7" event={"ID":"21c6b2e7-e755-4817-9284-606df07c2887","Type":"ContainerDied","Data":"a77b6c44ad541f9d1fa71bb0299842e46b5d41408dc1c45681bbe7a65e9c9fb3"} Sep 11 14:05:50.208981 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.208843 2519 scope.go:117] "RemoveContainer" containerID="d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934" Sep 11 14:05:50.210114 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.210094 2519 generic.go:356] "Generic (PLEG): container finished" podID="2282259a-a663-4e86-b50f-80edfc7e5c03" containerID="05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75" exitCode=0 Sep 11 14:05:50.210202 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.210134 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" Sep 11 14:05:50.210259 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.210135 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" event={"ID":"2282259a-a663-4e86-b50f-80edfc7e5c03","Type":"ContainerDied","Data":"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75"} Sep 11 14:05:50.210259 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.210236 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt" event={"ID":"2282259a-a663-4e86-b50f-80edfc7e5c03","Type":"ContainerDied","Data":"85b7ad58b905253eb7938802ac4b2564c65c3287a83d725451ce57f16d0d1872"} Sep 11 14:05:50.218390 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.218373 2519 scope.go:117] "RemoveContainer" containerID="d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934" Sep 11 14:05:50.218730 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:50.218708 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934\": container with ID starting with d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934 not found: ID does not exist" containerID="d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934" Sep 11 14:05:50.218797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.218737 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934"} err="failed to get container status \"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934\": rpc error: code = NotFound desc = could not find container \"d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934\": container with ID starting with d38f4469f8e93963b99792b5fd6cc79353e3b265709ead49a5dd31c822cdc934 not found: ID does not exist" Sep 11 14:05:50.218797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.218752 2519 scope.go:117] "RemoveContainer" containerID="05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75" Sep 11 14:05:50.226829 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.226812 2519 scope.go:117] "RemoveContainer" containerID="05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75" Sep 11 14:05:50.227086 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:50.227068 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75\": container with ID starting with 05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75 not found: ID does not exist" containerID="05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75" Sep 11 14:05:50.227147 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.227094 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75"} err="failed to get container status \"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75\": rpc error: code = NotFound desc = could not find container \"05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75\": container with ID starting with 05bc7ccf1bc746cf5890e97225d010ff5b3201a15fb47c506d072b572be23d75 not found: ID does not exist" Sep 11 14:05:50.235718 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.235699 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:05:50.240146 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.240125 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-da9534-5c58b4566c-sbsqt"] Sep 11 14:05:50.249860 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.249839 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:05:50.256129 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.256108 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-da9534-bbdd7f958-8d2x7"] Sep 11 14:05:50.691907 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.691872 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c6b2e7-e755-4817-9284-606df07c2887" path="/var/lib/kubelet/pods/21c6b2e7-e755-4817-9284-606df07c2887/volumes" Sep 11 14:05:50.692366 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.692347 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2282259a-a663-4e86-b50f-80edfc7e5c03" path="/var/lib/kubelet/pods/2282259a-a663-4e86-b50f-80edfc7e5c03/volumes" Sep 11 14:05:50.692779 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:50.692764 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47b04c8-3ba9-400a-89a2-b7208a3d4763" path="/var/lib/kubelet/pods/f47b04c8-3ba9-400a-89a2-b7208a3d4763/volumes" Sep 11 14:05:52.778524 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:52.778493 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:52.881527 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:52.881502 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.050833 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.050787 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.152254 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.152217 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.256166 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.256139 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.553323 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.553291 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.653099 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.653060 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:54.692955 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.692922 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:05:54.693320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693307 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47b04c8-3ba9-400a-89a2-b7208a3d4763" containerName="revoke-keys" Sep 11 14:05:54.693365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693321 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47b04c8-3ba9-400a-89a2-b7208a3d4763" containerName="revoke-keys" Sep 11 14:05:54.693365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693338 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2282259a-a663-4e86-b50f-80edfc7e5c03" containerName="payload-processing" Sep 11 14:05:54.693365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693345 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="2282259a-a663-4e86-b50f-80edfc7e5c03" containerName="payload-processing" Sep 11 14:05:54.693365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693353 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c6b2e7-e755-4817-9284-606df07c2887" containerName="payload-pre-processing" Sep 11 14:05:54.693365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693358 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c6b2e7-e755-4817-9284-606df07c2887" containerName="payload-pre-processing" Sep 11 14:05:54.693526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693412 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47b04c8-3ba9-400a-89a2-b7208a3d4763" containerName="revoke-keys" Sep 11 14:05:54.693526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693423 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c6b2e7-e755-4817-9284-606df07c2887" containerName="payload-pre-processing" Sep 11 14:05:54.693526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.693429 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="2282259a-a663-4e86-b50f-80edfc7e5c03" containerName="payload-processing" Sep 11 14:05:54.697773 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.697757 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.700103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.700081 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ffbdd858-dockercfg-sch4k\"" Sep 11 14:05:54.700227 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.700209 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ffbdd858\"" Sep 11 14:05:54.705994 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.705968 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:05:54.755145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.755106 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:05:54.758794 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.758776 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.768758 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.768731 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:05:54.801524 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.801494 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.801717 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.801555 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfc28\" (UniqueName: \"kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.801717 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.801633 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.801717 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.801709 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5d4\" (UniqueName: \"kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.903143 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.903102 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfc28\" (UniqueName: \"kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.903340 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.903165 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.903340 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.903192 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5d4\" (UniqueName: \"kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.903470 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.903353 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.903999 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.903975 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.904162 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.904137 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:54.910794 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.910764 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfc28\" (UniqueName: \"kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28\") pod \"payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:54.910914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:54.910771 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5d4\" (UniqueName: \"kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4\") pod \"payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:55.009245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.009205 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:55.071283 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.070978 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:55.164198 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.164110 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" podUID="576e5173-f39a-44f1-924f-ffff8d9ae836" containerName="main" containerID="cri-o://b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa" gracePeriod=45 Sep 11 14:05:55.280199 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.280172 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:05:55.282239 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:05:55.282203 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1997c7e7_b891_4e21_b0b7_c795578c475f.slice/crio-0562c8393971e5af8aa9b3cda4da2be0ed2a49c40a3cd091d38d1a292c3adb4b WatchSource:0}: Error finding container 0562c8393971e5af8aa9b3cda4da2be0ed2a49c40a3cd091d38d1a292c3adb4b: Status 404 returned error can't find the container with id 0562c8393971e5af8aa9b3cda4da2be0ed2a49c40a3cd091d38d1a292c3adb4b Sep 11 14:05:55.349159 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.349131 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:05:55.355337 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:05:55.355312 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe094b9_7a2f_42d6_b571_492e9e40e958.slice/crio-8197df07126a1bb75c6ed74f8fd69f0ce83d6ae742c9249ba67d349faecb2bd4 WatchSource:0}: Error finding container 8197df07126a1bb75c6ed74f8fd69f0ce83d6ae742c9249ba67d349faecb2bd4: Status 404 returned error can't find the container with id 8197df07126a1bb75c6ed74f8fd69f0ce83d6ae742c9249ba67d349faecb2bd4 Sep 11 14:05:55.426753 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.426713 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:55.508959 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.508922 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509153 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.508977 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42hcz\" (UniqueName: \"kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509153 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509097 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509153 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509148 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509188 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509205 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache" (OuterVolumeSpecName: "model-cache") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:55.509355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509215 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm\") pod \"576e5173-f39a-44f1-924f-ffff8d9ae836\" (UID: \"576e5173-f39a-44f1-924f-ffff8d9ae836\") " Sep 11 14:05:55.509678 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509654 2519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:55.509785 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509647 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home" (OuterVolumeSpecName: "home") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:55.509785 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.509773 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:55.511438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.511394 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz" (OuterVolumeSpecName: "kube-api-access-42hcz") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "kube-api-access-42hcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:55.511438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.511422 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm" (OuterVolumeSpecName: "dshm") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:55.511570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.511492 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "576e5173-f39a-44f1-924f-ffff8d9ae836" (UID: "576e5173-f39a-44f1-924f-ffff8d9ae836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:05:55.610763 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.610726 2519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576e5173-f39a-44f1-924f-ffff8d9ae836-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:55.610763 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.610755 2519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:55.610763 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.610764 2519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:55.610763 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.610771 2519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576e5173-f39a-44f1-924f-ffff8d9ae836-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:55.611031 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:55.610779 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hcz\" (UniqueName: \"kubernetes.io/projected/576e5173-f39a-44f1-924f-ffff8d9ae836-kube-api-access-42hcz\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:05:56.244990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.244887 2519 generic.go:356] "Generic (PLEG): container finished" podID="576e5173-f39a-44f1-924f-ffff8d9ae836" containerID="b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa" exitCode=0 Sep 11 14:05:56.244990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.244943 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" Sep 11 14:05:56.245516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.244983 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" event={"ID":"576e5173-f39a-44f1-924f-ffff8d9ae836","Type":"ContainerDied","Data":"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa"} Sep 11 14:05:56.245516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.245050 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5" event={"ID":"576e5173-f39a-44f1-924f-ffff8d9ae836","Type":"ContainerDied","Data":"02bfe134e44226b667dd253ba4662cd92d12acb01157dc4426612c06cf52d92e"} Sep 11 14:05:56.245516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.245077 2519 scope.go:117] "RemoveContainer" containerID="b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa" Sep 11 14:05:56.247075 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.247050 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" event={"ID":"ffe094b9-7a2f-42d6-b571-492e9e40e958","Type":"ContainerStarted","Data":"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c"} Sep 11 14:05:56.247183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.247134 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" event={"ID":"ffe094b9-7a2f-42d6-b571-492e9e40e958","Type":"ContainerStarted","Data":"8197df07126a1bb75c6ed74f8fd69f0ce83d6ae742c9249ba67d349faecb2bd4"} Sep 11 14:05:56.249171 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.249148 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" event={"ID":"1997c7e7-b891-4e21-b0b7-c795578c475f","Type":"ContainerStarted","Data":"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220"} Sep 11 14:05:56.249281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.249201 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:05:56.249281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.249226 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" event={"ID":"1997c7e7-b891-4e21-b0b7-c795578c475f","Type":"ContainerStarted","Data":"0562c8393971e5af8aa9b3cda4da2be0ed2a49c40a3cd091d38d1a292c3adb4b"} Sep 11 14:05:56.256619 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.256598 2519 scope.go:117] "RemoveContainer" containerID="b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa" Sep 11 14:05:56.256908 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:05:56.256889 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa\": container with ID starting with b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa not found: ID does not exist" containerID="b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa" Sep 11 14:05:56.256989 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.256918 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa"} err="failed to get container status \"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa\": rpc error: code = NotFound desc = could not find container \"b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa\": container with ID starting with b9f85e4e9ed0f526fcec068100deecad5bd0ad7ecb8b1141e1ae72db6444fdfa not found: ID does not exist" Sep 11 14:05:56.266711 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.266661 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" podStartSLOduration=2.2666437139999998 podStartE2EDuration="2.266643714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:56.265565617 +0000 UTC m=+1024.019880528" watchObservedRunningTime="2026-09-11 14:05:56.266643714 +0000 UTC m=+1024.020958608" Sep 11 14:05:56.279924 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.279898 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:56.285212 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.285192 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4edac4/noten-model-4edac4-kserve-84f5ff584c-fftn5"] Sep 11 14:05:56.298905 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.298847 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" podStartSLOduration=2.298830266 podStartE2EDuration="2.298830266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:56.297661802 +0000 UTC m=+1024.051976704" watchObservedRunningTime="2026-09-11 14:05:56.298830266 +0000 UTC m=+1024.053145157" Sep 11 14:05:56.455716 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.455683 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:56.554708 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.554621 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:56.694307 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.694268 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="576e5173-f39a-44f1-924f-ffff8d9ae836" path="/var/lib/kubelet/pods/576e5173-f39a-44f1-924f-ffff8d9ae836/volumes" Sep 11 14:05:56.703103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:56.703076 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:57.250208 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:57.250177 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:57.255771 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:57.255743 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:05:57.604818 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:57.604768 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:58.065142 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:58.065024 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:58.303515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:05:58.303453 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:01.151921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:01.151880 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:01.251887 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:01.251843 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:02.312042 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.312012 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:02.411815 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.411765 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:02.810952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.810917 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:02.910649 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.910595 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:02.925350 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.925318 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:06:02.925801 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.925789 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576e5173-f39a-44f1-924f-ffff8d9ae836" containerName="main" Sep 11 14:06:02.925862 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.925803 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="576e5173-f39a-44f1-924f-ffff8d9ae836" containerName="main" Sep 11 14:06:02.925899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.925878 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="576e5173-f39a-44f1-924f-ffff8d9ae836" containerName="main" Sep 11 14:06:02.930692 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.930662 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:02.933039 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.933017 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-932c97\"" Sep 11 14:06:02.933150 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.933047 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-932c97-dockercfg-ff4q6\"" Sep 11 14:06:02.942025 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.941999 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:06:02.951045 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.951014 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:06:02.955013 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.954991 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:02.970008 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.969980 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:06:02.987657 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.987622 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx488\" (UniqueName: \"kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:02.987842 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:02.987701 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.088435 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.088391 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.088668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.088458 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx488\" (UniqueName: \"kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.088668 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.088559 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.088814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.088709 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjm5t\" (UniqueName: \"kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.089225 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.089207 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.096226 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.096198 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx488\" (UniqueName: \"kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488\") pod \"payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.190040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.190000 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjm5t\" (UniqueName: \"kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.190237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.190061 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.190803 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.190778 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.197807 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.197785 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjm5t\" (UniqueName: \"kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t\") pod \"payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.242679 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.242634 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:03.270934 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.270900 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:03.526564 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.526538 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:06:03.528643 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:06:03.528612 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2738fd9_4003_47a8_a29b_a65d94414aaf.slice/crio-336a5b309b732b3b89935edf26f6113cd73db77aabe0576db63423b74f3836ba WatchSource:0}: Error finding container 336a5b309b732b3b89935edf26f6113cd73db77aabe0576db63423b74f3836ba: Status 404 returned error can't find the container with id 336a5b309b732b3b89935edf26f6113cd73db77aabe0576db63423b74f3836ba Sep 11 14:06:03.559933 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:03.559904 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:06:03.567822 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:06:03.567797 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aac3c2_c69c_4b0a_b08a_0ee5e490416f.slice/crio-d045d3cf8c3ef43b6df6e47523e90155a064a491373535ab3b3dac2eb23037af WatchSource:0}: Error finding container d045d3cf8c3ef43b6df6e47523e90155a064a491373535ab3b3dac2eb23037af: Status 404 returned error can't find the container with id d045d3cf8c3ef43b6df6e47523e90155a064a491373535ab3b3dac2eb23037af Sep 11 14:06:04.298357 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.298315 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" event={"ID":"86aac3c2-c69c-4b0a-b08a-0ee5e490416f","Type":"ContainerStarted","Data":"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4"} Sep 11 14:06:04.298600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.298381 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" event={"ID":"86aac3c2-c69c-4b0a-b08a-0ee5e490416f","Type":"ContainerStarted","Data":"d045d3cf8c3ef43b6df6e47523e90155a064a491373535ab3b3dac2eb23037af"} Sep 11 14:06:04.298600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.298411 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:04.300122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.300082 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" event={"ID":"f2738fd9-4003-47a8-a29b-a65d94414aaf","Type":"ContainerStarted","Data":"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105"} Sep 11 14:06:04.300211 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.300144 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" event={"ID":"f2738fd9-4003-47a8-a29b-a65d94414aaf","Type":"ContainerStarted","Data":"336a5b309b732b3b89935edf26f6113cd73db77aabe0576db63423b74f3836ba"} Sep 11 14:06:04.300262 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.300229 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:04.315350 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.315296 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" podStartSLOduration=2.315277168 podStartE2EDuration="2.315277168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:04.314762941 +0000 UTC m=+1032.069077829" watchObservedRunningTime="2026-09-11 14:06:04.315277168 +0000 UTC m=+1032.069592060" Sep 11 14:06:04.331555 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:04.331515 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" podStartSLOduration=2.331500762 podStartE2EDuration="2.331500762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:04.329811772 +0000 UTC m=+1032.084126663" watchObservedRunningTime="2026-09-11 14:06:04.331500762 +0000 UTC m=+1032.085815653" Sep 11 14:06:07.257837 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:07.257799 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:06:08.263686 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:08.263649 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:06:15.308040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:15.308009 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:06:15.308549 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:15.308081 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:06:16.047115 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:16.047079 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:17.798957 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:17.798920 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:18.461003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:18.460966 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:18.559784 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:18.559754 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:18.901659 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:18.901617 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:21.761507 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:21.761477 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:23.230963 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:23.230920 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:24.759942 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:24.759901 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:24.829007 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:24.828948 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:25.221715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:25.221681 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:26.094524 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:26.094481 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:26.426993 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:26.426862 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:26.737338 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:26.737250 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:26.885757 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:26.885725 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:29.436808 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:29.436774 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:29.526319 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:29.526283 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:30.758966 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:30.758882 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:30.860676 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:30.860644 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.269780 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.269718 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.377743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.377670 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.388250 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.388218 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:06:31.392254 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.392221 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.395243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.395220 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-367517\"" Sep 11 14:06:31.395368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.395263 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-367517-dockercfg-szgmf\"" Sep 11 14:06:31.400554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.400406 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:06:31.404620 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.404569 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:06:31.404749 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.404710 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.421460 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.421168 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:06:31.445094 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.445059 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbzt\" (UniqueName: \"kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.445265 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.445118 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2r5w\" (UniqueName: \"kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.445265 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.445167 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.445265 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.445218 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.546414 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.546328 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r5w\" (UniqueName: \"kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.546414 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.546370 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.546682 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.546431 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.546682 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.546522 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbzt\" (UniqueName: \"kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.547197 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.547168 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.547290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.547217 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.554191 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.554163 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbzt\" (UniqueName: \"kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt\") pod \"payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.554314 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.554265 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r5w\" (UniqueName: \"kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w\") pod \"payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.704634 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.704595 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:31.716960 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.716933 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:31.861810 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.861773 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.980038 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:31.980009 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:06:31.981588 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:06:31.981543 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816f1ee5_762f_473f_928b_fd9baf9e4e47.slice/crio-11e761a836c7c38ec081d888b35ea254b181034513b5cf9a9ac47c75a5a3722c WatchSource:0}: Error finding container 11e761a836c7c38ec081d888b35ea254b181034513b5cf9a9ac47c75a5a3722c: Status 404 returned error can't find the container with id 11e761a836c7c38ec081d888b35ea254b181034513b5cf9a9ac47c75a5a3722c Sep 11 14:06:32.008127 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.008102 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:06:32.014778 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:06:32.014752 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e6f873_510d_4c49_b729_2591508bed52.slice/crio-800bc0324d0be86dae8d588bd1755e541f0e13a0c744ac6feb7a1536d6fe534e WatchSource:0}: Error finding container 800bc0324d0be86dae8d588bd1755e541f0e13a0c744ac6feb7a1536d6fe534e: Status 404 returned error can't find the container with id 800bc0324d0be86dae8d588bd1755e541f0e13a0c744ac6feb7a1536d6fe534e Sep 11 14:06:32.462031 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.461992 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" event={"ID":"45e6f873-510d-4c49-b729-2591508bed52","Type":"ContainerStarted","Data":"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461"} Sep 11 14:06:32.462243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.462056 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" event={"ID":"45e6f873-510d-4c49-b729-2591508bed52","Type":"ContainerStarted","Data":"800bc0324d0be86dae8d588bd1755e541f0e13a0c744ac6feb7a1536d6fe534e"} Sep 11 14:06:32.462243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.462194 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:32.464012 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.463981 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" event={"ID":"816f1ee5-762f-473f-928b-fd9baf9e4e47","Type":"ContainerStarted","Data":"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911"} Sep 11 14:06:32.464152 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.464037 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" event={"ID":"816f1ee5-762f-473f-928b-fd9baf9e4e47","Type":"ContainerStarted","Data":"11e761a836c7c38ec081d888b35ea254b181034513b5cf9a9ac47c75a5a3722c"} Sep 11 14:06:32.464152 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.464065 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:32.479756 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.479702 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" podStartSLOduration=1.4796851549999999 podStartE2EDuration="1.479685155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:32.477460577 +0000 UTC m=+1060.231775471" watchObservedRunningTime="2026-09-11 14:06:32.479685155 +0000 UTC m=+1060.234000045" Sep 11 14:06:32.495011 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:32.494958 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" podStartSLOduration=1.494938672 podStartE2EDuration="1.494938672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:32.49221925 +0000 UTC m=+1060.246534142" watchObservedRunningTime="2026-09-11 14:06:32.494938672 +0000 UTC m=+1060.249253562" Sep 11 14:06:35.252748 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:35.252715 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.025870 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.025836 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.054903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.054866 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:06:36.055410 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.055359 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" podUID="1997c7e7-b891-4e21-b0b7-c795578c475f" containerName="payload-processing" containerID="cri-o://975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220" gracePeriod=30 Sep 11 14:06:36.065743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.065710 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:06:36.066004 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.065953 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" podUID="ffe094b9-7a2f-42d6-b571-492e9e40e958" containerName="payload-pre-processing" containerID="cri-o://383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c" gracePeriod=30 Sep 11 14:06:36.100442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.100408 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.165723 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.165693 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.245745 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.245707 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.339714 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.339688 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:06:36.342486 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.342470 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:06:36.351371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.351351 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.395125 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395093 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfc28\" (UniqueName: \"kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28\") pod \"1997c7e7-b891-4e21-b0b7-c795578c475f\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " Sep 11 14:06:36.395291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395149 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume\") pod \"ffe094b9-7a2f-42d6-b571-492e9e40e958\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " Sep 11 14:06:36.395291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395214 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume\") pod \"1997c7e7-b891-4e21-b0b7-c795578c475f\" (UID: \"1997c7e7-b891-4e21-b0b7-c795578c475f\") " Sep 11 14:06:36.395291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395265 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5d4\" (UniqueName: \"kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4\") pod \"ffe094b9-7a2f-42d6-b571-492e9e40e958\" (UID: \"ffe094b9-7a2f-42d6-b571-492e9e40e958\") " Sep 11 14:06:36.395662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395630 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1997c7e7-b891-4e21-b0b7-c795578c475f" (UID: "1997c7e7-b891-4e21-b0b7-c795578c475f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:06:36.395662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.395649 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffe094b9-7a2f-42d6-b571-492e9e40e958" (UID: "ffe094b9-7a2f-42d6-b571-492e9e40e958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:06:36.397405 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.397378 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28" (OuterVolumeSpecName: "kube-api-access-bfc28") pod "1997c7e7-b891-4e21-b0b7-c795578c475f" (UID: "1997c7e7-b891-4e21-b0b7-c795578c475f"). InnerVolumeSpecName "kube-api-access-bfc28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:06:36.397509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.397460 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4" (OuterVolumeSpecName: "kube-api-access-kc5d4") pod "ffe094b9-7a2f-42d6-b571-492e9e40e958" (UID: "ffe094b9-7a2f-42d6-b571-492e9e40e958"). InnerVolumeSpecName "kube-api-access-kc5d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:06:36.489223 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.489187 2519 generic.go:356] "Generic (PLEG): container finished" podID="ffe094b9-7a2f-42d6-b571-492e9e40e958" containerID="383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c" exitCode=0 Sep 11 14:06:36.489397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.489236 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" Sep 11 14:06:36.489397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.489283 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" event={"ID":"ffe094b9-7a2f-42d6-b571-492e9e40e958","Type":"ContainerDied","Data":"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c"} Sep 11 14:06:36.489397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.489347 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n" event={"ID":"ffe094b9-7a2f-42d6-b571-492e9e40e958","Type":"ContainerDied","Data":"8197df07126a1bb75c6ed74f8fd69f0ce83d6ae742c9249ba67d349faecb2bd4"} Sep 11 14:06:36.489397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.489384 2519 scope.go:117] "RemoveContainer" containerID="383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c" Sep 11 14:06:36.490862 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.490835 2519 generic.go:356] "Generic (PLEG): container finished" podID="1997c7e7-b891-4e21-b0b7-c795578c475f" containerID="975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220" exitCode=0 Sep 11 14:06:36.490959 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.490870 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" event={"ID":"1997c7e7-b891-4e21-b0b7-c795578c475f","Type":"ContainerDied","Data":"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220"} Sep 11 14:06:36.490959 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.490882 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" Sep 11 14:06:36.490959 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.490912 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx" event={"ID":"1997c7e7-b891-4e21-b0b7-c795578c475f","Type":"ContainerDied","Data":"0562c8393971e5af8aa9b3cda4da2be0ed2a49c40a3cd091d38d1a292c3adb4b"} Sep 11 14:06:36.496257 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.496232 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfc28\" (UniqueName: \"kubernetes.io/projected/1997c7e7-b891-4e21-b0b7-c795578c475f-kube-api-access-bfc28\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:06:36.496344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.496262 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe094b9-7a2f-42d6-b571-492e9e40e958-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:06:36.496344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.496277 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1997c7e7-b891-4e21-b0b7-c795578c475f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:06:36.496344 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.496291 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5d4\" (UniqueName: \"kubernetes.io/projected/ffe094b9-7a2f-42d6-b571-492e9e40e958-kube-api-access-kc5d4\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:06:36.500732 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.500584 2519 scope.go:117] "RemoveContainer" containerID="383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c" Sep 11 14:06:36.501139 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:06:36.501111 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c\": container with ID starting with 383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c not found: ID does not exist" containerID="383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c" Sep 11 14:06:36.501232 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.501148 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c"} err="failed to get container status \"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c\": rpc error: code = NotFound desc = could not find container \"383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c\": container with ID starting with 383158b91a116dc3a6fc8f3c01602513f85d377be41754c1043eef23f0049d4c not found: ID does not exist" Sep 11 14:06:36.501232 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.501173 2519 scope.go:117] "RemoveContainer" containerID="975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220" Sep 11 14:06:36.509918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.509897 2519 scope.go:117] "RemoveContainer" containerID="975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220" Sep 11 14:06:36.510219 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:06:36.510200 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220\": container with ID starting with 975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220 not found: ID does not exist" containerID="975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220" Sep 11 14:06:36.510295 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.510232 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220"} err="failed to get container status \"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220\": rpc error: code = NotFound desc = could not find container \"975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220\": container with ID starting with 975b6bea48b2d16d4d36e027307386183b0f699a4d94c44652a75b9ac1ae7220 not found: ID does not exist" Sep 11 14:06:36.514288 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.514264 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:06:36.521021 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.520998 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ffbdd858-599b557dfd-p8l9n"] Sep 11 14:06:36.530171 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.530146 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:06:36.535074 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.535051 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ffbdd858-56b785f67c-26zxx"] Sep 11 14:06:36.603599 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.603548 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.693395 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.693359 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1997c7e7-b891-4e21-b0b7-c795578c475f" path="/var/lib/kubelet/pods/1997c7e7-b891-4e21-b0b7-c795578c475f/volumes" Sep 11 14:06:36.693823 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:36.693807 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffe094b9-7a2f-42d6-b571-492e9e40e958" path="/var/lib/kubelet/pods/ffe094b9-7a2f-42d6-b571-492e9e40e958/volumes" Sep 11 14:06:42.384218 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:42.384181 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:42.456380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:42.456347 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:42.645513 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:42.645426 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.006299 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:43.006180 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.096172 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:43.096135 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.200195 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:43.200160 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.471769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:43.471739 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:06:43.472217 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:43.471821 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:06:49.109240 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:49.109165 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.212239 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:49.212205 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.308364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:49.308330 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.467140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:49.467054 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:51.751400 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:51.751363 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.342196 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:54.342161 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:55.460972 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:55.460936 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:55.533790 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:55.533757 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.401506 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:56.401468 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.710729 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:56.710643 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.878833 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:06:56.878800 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:00.152101 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:00.152065 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:00.251593 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:00.251551 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.314734 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.314692 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.415413 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.415375 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.514486 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.514456 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.820515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.820484 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.924980 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.924950 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.932292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932257 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:07:01.932818 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932802 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe094b9-7a2f-42d6-b571-492e9e40e958" containerName="payload-pre-processing" Sep 11 14:07:01.932875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932821 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe094b9-7a2f-42d6-b571-492e9e40e958" containerName="payload-pre-processing" Sep 11 14:07:01.932875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932840 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1997c7e7-b891-4e21-b0b7-c795578c475f" containerName="payload-processing" Sep 11 14:07:01.932875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932846 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="1997c7e7-b891-4e21-b0b7-c795578c475f" containerName="payload-processing" Sep 11 14:07:01.932969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932910 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe094b9-7a2f-42d6-b571-492e9e40e958" containerName="payload-pre-processing" Sep 11 14:07:01.932969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.932921 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="1997c7e7-b891-4e21-b0b7-c795578c475f" containerName="payload-processing" Sep 11 14:07:01.937909 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.937892 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:01.940044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.940019 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6b9470\"" Sep 11 14:07:01.940180 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.940160 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6b9470-dockercfg-9xsqf\"" Sep 11 14:07:01.950356 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.950332 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:07:01.960604 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.960539 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:07:01.964456 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.964427 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:01.972741 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:01.972715 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:07:02.048147 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.048107 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjbm\" (UniqueName: \"kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.048348 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.048271 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.149191 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.149146 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjbm\" (UniqueName: \"kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.149387 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.149221 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.149387 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.149248 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9mc\" (UniqueName: \"kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.149498 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.149399 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.150123 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.150097 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.165142 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.165111 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjbm\" (UniqueName: \"kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm\") pod \"payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.250221 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.250189 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.250435 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.250401 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.250512 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.250446 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9mc\" (UniqueName: \"kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.251220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.251199 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.257961 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.257937 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9mc\" (UniqueName: \"kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc\") pod \"payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.277960 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.277931 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:02.532602 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.526659 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:07:02.558685 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.558663 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:07:02.567797 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:02.567759 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41119af7_58d4_4203_8438_933fccd87d6f.slice/crio-049c5313344e28b99a4be0525a6b462da35bbcbbc8898d858e67080f5af21966 WatchSource:0}: Error finding container 049c5313344e28b99a4be0525a6b462da35bbcbbc8898d858e67080f5af21966: Status 404 returned error can't find the container with id 049c5313344e28b99a4be0525a6b462da35bbcbbc8898d858e67080f5af21966 Sep 11 14:07:02.642496 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.642463 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" event={"ID":"28be26a6-72be-410a-a933-cfa5b2aa1c4d","Type":"ContainerStarted","Data":"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1"} Sep 11 14:07:02.642651 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.642523 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" event={"ID":"28be26a6-72be-410a-a933-cfa5b2aa1c4d","Type":"ContainerStarted","Data":"84599eb7a2ce0605c44058c3c9647406b97f9b6c23bdd2dfe0b23670230e830a"} Sep 11 14:07:02.642651 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.642624 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:02.644508 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.644480 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" event={"ID":"41119af7-58d4-4203-8438-933fccd87d6f","Type":"ContainerStarted","Data":"049c5313344e28b99a4be0525a6b462da35bbcbbc8898d858e67080f5af21966"} Sep 11 14:07:02.658847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:02.658793 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" podStartSLOduration=1.658773473 podStartE2EDuration="1.658773473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:02.656953768 +0000 UTC m=+1090.411268656" watchObservedRunningTime="2026-09-11 14:07:02.658773473 +0000 UTC m=+1090.413088363" Sep 11 14:07:03.652523 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:03.652480 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" event={"ID":"41119af7-58d4-4203-8438-933fccd87d6f","Type":"ContainerStarted","Data":"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919"} Sep 11 14:07:03.652972 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:03.652705 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:03.668788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:03.668743 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" podStartSLOduration=2.668728283 podStartE2EDuration="2.668728283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:03.667584467 +0000 UTC m=+1091.421899350" watchObservedRunningTime="2026-09-11 14:07:03.668728283 +0000 UTC m=+1091.423043173" Sep 11 14:07:13.654271 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:13.654234 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:07:14.660222 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:14.660189 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:07:19.211656 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:19.211612 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:25.109080 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:25.109047 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:25.444419 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:25.444321 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:25.754643 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:25.754513 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:25.907636 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:25.907603 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:28.744399 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:28.744357 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:28.823154 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:28.823117 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:29.954272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:29.954236 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:30.055766 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.055726 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:30.471053 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.471015 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:30.568814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.568769 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:30.608263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.608232 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:07:30.611910 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.611891 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.614274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.614250 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-01faa9-dockercfg-5nvtx\"" Sep 11 14:07:30.614387 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.614291 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-01faa9\"" Sep 11 14:07:30.650252 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.650217 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:07:30.653836 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.653814 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.662059 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.661320 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:07:30.664612 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.664587 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:07:30.716970 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.716883 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.716970 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.716944 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xbd\" (UniqueName: \"kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.717195 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.717054 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.717195 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.717126 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znzx\" (UniqueName: \"kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.817988 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.817955 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znzx\" (UniqueName: \"kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.818203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.818026 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.818203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.818052 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xbd\" (UniqueName: \"kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.818203 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.818100 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.818847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.818822 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.818956 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.818842 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.825826 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.825801 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znzx\" (UniqueName: \"kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx\") pod \"payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.825945 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.825918 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xbd\" (UniqueName: \"kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd\") pod \"payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.923353 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.923320 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:30.965781 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.965745 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:30.984556 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:30.984500 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:31.203758 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.203732 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:07:31.205091 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:31.205064 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619dcec7_56c1_45a8_9bed_b89435d1d652.slice/crio-f22270e89979e5c80e9359826aca57f7cb55db6416c4b72e28574472f9d2d73b WatchSource:0}: Error finding container f22270e89979e5c80e9359826aca57f7cb55db6416c4b72e28574472f9d2d73b: Status 404 returned error can't find the container with id f22270e89979e5c80e9359826aca57f7cb55db6416c4b72e28574472f9d2d73b Sep 11 14:07:31.254076 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.254049 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:07:31.255538 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:31.255515 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4561b9a_3861_443f_9fac_04c21591c3ed.slice/crio-65dc9ea58ee7fcfc3a6b7060390435dbed33d48e84ec77ee73a5df57300de8d5 WatchSource:0}: Error finding container 65dc9ea58ee7fcfc3a6b7060390435dbed33d48e84ec77ee73a5df57300de8d5: Status 404 returned error can't find the container with id 65dc9ea58ee7fcfc3a6b7060390435dbed33d48e84ec77ee73a5df57300de8d5 Sep 11 14:07:31.811868 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.811821 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" event={"ID":"619dcec7-56c1-45a8-9bed-b89435d1d652","Type":"ContainerStarted","Data":"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec"} Sep 11 14:07:31.812070 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.811884 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" event={"ID":"619dcec7-56c1-45a8-9bed-b89435d1d652","Type":"ContainerStarted","Data":"f22270e89979e5c80e9359826aca57f7cb55db6416c4b72e28574472f9d2d73b"} Sep 11 14:07:31.812070 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.811914 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:31.813490 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.813462 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" event={"ID":"f4561b9a-3861-443f-9fac-04c21591c3ed","Type":"ContainerStarted","Data":"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27"} Sep 11 14:07:31.813627 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.813514 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" event={"ID":"f4561b9a-3861-443f-9fac-04c21591c3ed","Type":"ContainerStarted","Data":"65dc9ea58ee7fcfc3a6b7060390435dbed33d48e84ec77ee73a5df57300de8d5"} Sep 11 14:07:31.813627 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.813539 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:31.829185 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.829134 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" podStartSLOduration=1.829114794 podStartE2EDuration="1.829114794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:31.826262688 +0000 UTC m=+1119.580577578" watchObservedRunningTime="2026-09-11 14:07:31.829114794 +0000 UTC m=+1119.583429685" Sep 11 14:07:31.844271 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:31.844229 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" podStartSLOduration=1.84421416 podStartE2EDuration="1.84421416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:31.842896552 +0000 UTC m=+1119.597211442" watchObservedRunningTime="2026-09-11 14:07:31.84421416 +0000 UTC m=+1119.598529061" Sep 11 14:07:33.656621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:33.656588 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.756856 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:33.756824 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:42.821049 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:42.821014 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:07:42.821427 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:42.821113 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:07:51.600363 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:51.600326 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:53.721494 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:53.721459 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:54.522451 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.522416 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7540-28d5a7"] Sep 11 14:07:54.526000 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.525952 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:07:54.528155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.528135 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qcb7\"" Sep 11 14:07:54.534773 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.534740 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-28d5a7"] Sep 11 14:07:54.550568 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.550535 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2jq\" (UniqueName: \"kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq\") pod \"test-curl-7540-28d5a7\" (UID: \"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0\") " pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:07:54.642632 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.642568 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:54.651322 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.651290 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2jq\" (UniqueName: \"kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq\") pod \"test-curl-7540-28d5a7\" (UID: \"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0\") " pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:07:54.659041 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.659017 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2jq\" (UniqueName: \"kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq\") pod \"test-curl-7540-28d5a7\" (UID: \"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0\") " pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:07:54.837163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:54.837133 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:07:55.106698 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:55.106670 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-28d5a7"] Sep 11 14:07:55.108442 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:55.108410 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c869d0d_9705_4b93_b05c_71f3dd3e8dc0.slice/crio-a371094c2ca341b4e1c7eeb40cfe600083f866824469e389b9aad01a29255a40 WatchSource:0}: Error finding container a371094c2ca341b4e1c7eeb40cfe600083f866824469e389b9aad01a29255a40: Status 404 returned error can't find the container with id a371094c2ca341b4e1c7eeb40cfe600083f866824469e389b9aad01a29255a40 Sep 11 14:07:55.959665 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:55.959621 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-28d5a7" event={"ID":"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0","Type":"ContainerStarted","Data":"164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0"} Sep 11 14:07:55.960249 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:55.960223 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-28d5a7" event={"ID":"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0","Type":"ContainerStarted","Data":"a371094c2ca341b4e1c7eeb40cfe600083f866824469e389b9aad01a29255a40"} Sep 11 14:07:55.976638 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:55.976548 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7540-28d5a7" podStartSLOduration=1.97653275 podStartE2EDuration="1.97653275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:55.974327543 +0000 UTC m=+1143.728642433" watchObservedRunningTime="2026-09-11 14:07:55.97653275 +0000 UTC m=+1143.730847641" Sep 11 14:07:56.593493 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.593452 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7540-28d5a7"] Sep 11 14:07:56.599783 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.599749 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7540-28d5a7"] Sep 11 14:07:56.719636 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.719547 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7540-3b7205"] Sep 11 14:07:56.720034 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.720020 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" containerName="test-curl-7540-28d5a7" Sep 11 14:07:56.720079 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.720036 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" containerName="test-curl-7540-28d5a7" Sep 11 14:07:56.720122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.720113 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" containerName="test-curl-7540-28d5a7" Sep 11 14:07:56.723485 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.723466 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:07:56.730274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.730244 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-3b7205"] Sep 11 14:07:56.770014 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.769979 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slk7x\" (UniqueName: \"kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x\") pod \"test-curl-7540-3b7205\" (UID: \"41bc8678-f408-4cbb-85f8-ac04c90ddb6a\") " pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:07:56.870917 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.870874 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slk7x\" (UniqueName: \"kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x\") pod \"test-curl-7540-3b7205\" (UID: \"41bc8678-f408-4cbb-85f8-ac04c90ddb6a\") " pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:07:56.879621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:56.879566 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slk7x\" (UniqueName: \"kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x\") pod \"test-curl-7540-3b7205\" (UID: \"41bc8678-f408-4cbb-85f8-ac04c90ddb6a\") " pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:07:56.966020 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:07:56.965984 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7540-28d5a7" container="test-curl-7540-28d5a7" Sep 11 14:07:57.035300 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.035215 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:07:57.308163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.308138 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-3b7205"] Sep 11 14:07:57.310037 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:57.310012 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41bc8678_f408_4cbb_85f8_ac04c90ddb6a.slice/crio-1245440438d880b736cbaff7a4e8f8a73409fdd60d43bb56d650015d51ace467 WatchSource:0}: Error finding container 1245440438d880b736cbaff7a4e8f8a73409fdd60d43bb56d650015d51ace467: Status 404 returned error can't find the container with id 1245440438d880b736cbaff7a4e8f8a73409fdd60d43bb56d650015d51ace467 Sep 11 14:07:57.973124 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.973087 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-3b7205" event={"ID":"41bc8678-f408-4cbb-85f8-ac04c90ddb6a","Type":"ContainerStarted","Data":"32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b"} Sep 11 14:07:57.973614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.973132 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-3b7205" event={"ID":"41bc8678-f408-4cbb-85f8-ac04c90ddb6a","Type":"ContainerStarted","Data":"1245440438d880b736cbaff7a4e8f8a73409fdd60d43bb56d650015d51ace467"} Sep 11 14:07:57.973614 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.973218 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7540-28d5a7" podUID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" containerName="test-curl-7540-28d5a7" containerID="cri-o://164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0" gracePeriod=2 Sep 11 14:07:57.987957 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:57.987905 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7540-3b7205" podStartSLOduration=1.9878870910000002 podStartE2EDuration="1.987887091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:57.985736193 +0000 UTC m=+1145.740051085" watchObservedRunningTime="2026-09-11 14:07:57.987887091 +0000 UTC m=+1145.742201982" Sep 11 14:07:58.602268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.602226 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7540-3b7205"] Sep 11 14:07:58.606129 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.606098 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7540-3b7205"] Sep 11 14:07:58.722900 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.722820 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7540-0d7fd4"] Sep 11 14:07:58.723284 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.723270 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" containerName="test-curl-7540-3b7205" Sep 11 14:07:58.723331 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.723286 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" containerName="test-curl-7540-3b7205" Sep 11 14:07:58.723364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.723357 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" containerName="test-curl-7540-3b7205" Sep 11 14:07:58.726649 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.726630 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:07:58.730561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.730534 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-0d7fd4"] Sep 11 14:07:58.790820 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.790780 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wjw\" (UniqueName: \"kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw\") pod \"test-curl-7540-0d7fd4\" (UID: \"b1f56f8f-fbab-4748-8021-ac3aede01234\") " pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:07:58.891896 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.891860 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wjw\" (UniqueName: \"kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw\") pod \"test-curl-7540-0d7fd4\" (UID: \"b1f56f8f-fbab-4748-8021-ac3aede01234\") " pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:07:58.900140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:58.900115 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wjw\" (UniqueName: \"kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw\") pod \"test-curl-7540-0d7fd4\" (UID: \"b1f56f8f-fbab-4748-8021-ac3aede01234\") " pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:07:58.981265 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:07:58.981169 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7540-3b7205" container="test-curl-7540-3b7205" Sep 11 14:07:59.038983 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:59.038946 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:07:59.311423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:59.311399 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-0d7fd4"] Sep 11 14:07:59.313322 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:07:59.313295 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f56f8f_fbab_4748_8021_ac3aede01234.slice/crio-f9ac293577fc0e91e580801965df44e5ef9da2f44670bd50fc8b2384a4a180bc WatchSource:0}: Error finding container f9ac293577fc0e91e580801965df44e5ef9da2f44670bd50fc8b2384a4a180bc: Status 404 returned error can't find the container with id f9ac293577fc0e91e580801965df44e5ef9da2f44670bd50fc8b2384a4a180bc Sep 11 14:07:59.988940 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:59.988911 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-0d7fd4" event={"ID":"b1f56f8f-fbab-4748-8021-ac3aede01234","Type":"ContainerStarted","Data":"ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93"} Sep 11 14:07:59.989304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:59.988955 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-0d7fd4" event={"ID":"b1f56f8f-fbab-4748-8021-ac3aede01234","Type":"ContainerStarted","Data":"f9ac293577fc0e91e580801965df44e5ef9da2f44670bd50fc8b2384a4a180bc"} Sep 11 14:07:59.989304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:07:59.988953 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7540-3b7205" podUID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" containerName="test-curl-7540-3b7205" containerID="cri-o://32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b" gracePeriod=2 Sep 11 14:08:00.002896 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.002847 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7540-0d7fd4" podStartSLOduration=2.002829387 podStartE2EDuration="2.002829387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:00.001677907 +0000 UTC m=+1147.755992799" watchObservedRunningTime="2026-09-11 14:08:00.002829387 +0000 UTC m=+1147.757144279" Sep 11 14:08:00.172421 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.172396 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:08:00.203729 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.203696 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg2jq\" (UniqueName: \"kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq\") pod \"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0\" (UID: \"7c869d0d-9705-4b93-b05c-71f3dd3e8dc0\") " Sep 11 14:08:00.205738 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.205707 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq" (OuterVolumeSpecName: "kube-api-access-wg2jq") pod "7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" (UID: "7c869d0d-9705-4b93-b05c-71f3dd3e8dc0"). InnerVolumeSpecName "kube-api-access-wg2jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:00.304793 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.304703 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2jq\" (UniqueName: \"kubernetes.io/projected/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0-kube-api-access-wg2jq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:08:00.621051 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.621010 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7540-0d7fd4"] Sep 11 14:08:00.625655 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.625624 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7540-0d7fd4"] Sep 11 14:08:00.691557 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.691518 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" path="/var/lib/kubelet/pods/7c869d0d-9705-4b93-b05c-71f3dd3e8dc0/volumes" Sep 11 14:08:00.995607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.995491 2519 generic.go:356] "Generic (PLEG): container finished" podID="7c869d0d-9705-4b93-b05c-71f3dd3e8dc0" containerID="164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0" exitCode=137 Sep 11 14:08:00.995607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.995538 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-28d5a7" Sep 11 14:08:00.996093 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:00.995612 2519 scope.go:117] "RemoveContainer" containerID="164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0" Sep 11 14:08:00.996093 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:08:00.995792 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7540-0d7fd4" container="test-curl-7540-0d7fd4" Sep 11 14:08:01.004395 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:01.004381 2519 scope.go:117] "RemoveContainer" containerID="164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0" Sep 11 14:08:01.004680 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:08:01.004661 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0\": container with ID starting with 164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0 not found: ID does not exist" containerID="164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0" Sep 11 14:08:01.004729 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:01.004689 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0"} err="failed to get container status \"164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0\": rpc error: code = NotFound desc = could not find container \"164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0\": container with ID starting with 164950e83f91a1c87ae14ba0e4c16a40d59872214b0e9e58db1d11ed289311d0 not found: ID does not exist" Sep 11 14:08:02.003859 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.003828 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7540-0d7fd4" podUID="b1f56f8f-fbab-4748-8021-ac3aede01234" containerName="test-curl-7540-0d7fd4" containerID="cri-o://ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93" gracePeriod=2 Sep 11 14:08:02.173004 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.172977 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:08:02.222956 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.222923 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slk7x\" (UniqueName: \"kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x\") pod \"41bc8678-f408-4cbb-85f8-ac04c90ddb6a\" (UID: \"41bc8678-f408-4cbb-85f8-ac04c90ddb6a\") " Sep 11 14:08:02.224987 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.224950 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x" (OuterVolumeSpecName: "kube-api-access-slk7x") pod "41bc8678-f408-4cbb-85f8-ac04c90ddb6a" (UID: "41bc8678-f408-4cbb-85f8-ac04c90ddb6a"). InnerVolumeSpecName "kube-api-access-slk7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:02.324178 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.324097 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slk7x\" (UniqueName: \"kubernetes.io/projected/41bc8678-f408-4cbb-85f8-ac04c90ddb6a-kube-api-access-slk7x\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:08:02.691955 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:02.691921 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" path="/var/lib/kubelet/pods/41bc8678-f408-4cbb-85f8-ac04c90ddb6a/volumes" Sep 11 14:08:03.011430 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:03.011333 2519 generic.go:356] "Generic (PLEG): container finished" podID="41bc8678-f408-4cbb-85f8-ac04c90ddb6a" containerID="32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b" exitCode=137 Sep 11 14:08:03.011430 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:03.011391 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-3b7205" Sep 11 14:08:03.011430 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:03.011417 2519 scope.go:117] "RemoveContainer" containerID="32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b" Sep 11 14:08:03.020509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:03.020488 2519 scope.go:117] "RemoveContainer" containerID="32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b" Sep 11 14:08:03.020810 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:08:03.020786 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b\": container with ID starting with 32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b not found: ID does not exist" containerID="32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b" Sep 11 14:08:03.020900 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:03.020817 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b"} err="failed to get container status \"32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b\": rpc error: code = NotFound desc = could not find container \"32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b\": container with ID starting with 32705fa2f7557d27005e9459253727084b23e26d94e64e9e1a75fd396231037b not found: ID does not exist" Sep 11 14:08:04.196983 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.196951 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:08:04.247202 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.247170 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8wjw\" (UniqueName: \"kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw\") pod \"b1f56f8f-fbab-4748-8021-ac3aede01234\" (UID: \"b1f56f8f-fbab-4748-8021-ac3aede01234\") " Sep 11 14:08:04.249228 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.249203 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw" (OuterVolumeSpecName: "kube-api-access-f8wjw") pod "b1f56f8f-fbab-4748-8021-ac3aede01234" (UID: "b1f56f8f-fbab-4748-8021-ac3aede01234"). InnerVolumeSpecName "kube-api-access-f8wjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:04.295480 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.295400 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:04.348355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.348311 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wjw\" (UniqueName: \"kubernetes.io/projected/b1f56f8f-fbab-4748-8021-ac3aede01234-kube-api-access-f8wjw\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:08:04.609871 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.609837 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:04.691806 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:04.691772 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f56f8f-fbab-4748-8021-ac3aede01234" path="/var/lib/kubelet/pods/b1f56f8f-fbab-4748-8021-ac3aede01234/volumes" Sep 11 14:08:05.026742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.026634 2519 generic.go:356] "Generic (PLEG): container finished" podID="b1f56f8f-fbab-4748-8021-ac3aede01234" containerID="ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93" exitCode=137 Sep 11 14:08:05.026742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.026682 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-0d7fd4" Sep 11 14:08:05.026742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.026734 2519 scope.go:117] "RemoveContainer" containerID="ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93" Sep 11 14:08:05.035479 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.035460 2519 scope.go:117] "RemoveContainer" containerID="ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93" Sep 11 14:08:05.035773 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:08:05.035755 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93\": container with ID starting with ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93 not found: ID does not exist" containerID="ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93" Sep 11 14:08:05.035833 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.035781 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93"} err="failed to get container status \"ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93\": rpc error: code = NotFound desc = could not find container \"ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93\": container with ID starting with ef4384e039b5fc411e91925e2f2cedd50fa39fffcb9b65dcee84561be3d43f93 not found: ID does not exist" Sep 11 14:08:05.372921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:05.372883 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.021625 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:08.021591 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.111172 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:08.111143 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.198290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.198257 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.294442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.294408 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.393455 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.393424 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.714750 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.714715 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.808210 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.808173 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.860197 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.860155 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:08:09.860715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.860696 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f56f8f-fbab-4748-8021-ac3aede01234" containerName="test-curl-7540-0d7fd4" Sep 11 14:08:09.860715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.860716 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f56f8f-fbab-4748-8021-ac3aede01234" containerName="test-curl-7540-0d7fd4" Sep 11 14:08:09.860898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.860884 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f56f8f-fbab-4748-8021-ac3aede01234" containerName="test-curl-7540-0d7fd4" Sep 11 14:08:09.869553 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.869524 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:09.871986 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.871963 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc83333c\"" Sep 11 14:08:09.872153 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.872031 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc83333c-dockercfg-fth6n\"" Sep 11 14:08:09.873443 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.873402 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:08:09.899997 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.899969 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:08:09.903617 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.903600 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:09.914018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:09.913992 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:08:10.005393 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.005297 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.005561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.005393 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknxs\" (UniqueName: \"kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.005561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.005447 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.005561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.005483 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.106847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.106802 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.107059 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.106936 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.107059 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.106985 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknxs\" (UniqueName: \"kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.107059 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.107029 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.107612 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.107546 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.107782 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.107761 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.115169 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.115144 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknxs\" (UniqueName: \"kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs\") pod \"payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.115380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.115347 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg\") pod \"payload-processing-e2e-mt-fc83333c-b557d67df-hrf48\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.181558 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.181508 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:10.215815 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.215776 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:10.461254 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.461228 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:08:10.464137 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:08:10.464103 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5d284a_4668_4379_931d_7aa4c1fd0122.slice/crio-b855d645c8804c6c00a1c74608fb9e4bf076fb37310419b8edb5bce2239a8385 WatchSource:0}: Error finding container b855d645c8804c6c00a1c74608fb9e4bf076fb37310419b8edb5bce2239a8385: Status 404 returned error can't find the container with id b855d645c8804c6c00a1c74608fb9e4bf076fb37310419b8edb5bce2239a8385 Sep 11 14:08:10.499903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:10.499872 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:08:10.502321 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:08:10.502292 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f810bc_f6f5_437a_a28f_174643e32278.slice/crio-c7e5cecea863a0015cea392d853216d04c1ed5348839d3d29b083918867502ad WatchSource:0}: Error finding container c7e5cecea863a0015cea392d853216d04c1ed5348839d3d29b083918867502ad: Status 404 returned error can't find the container with id c7e5cecea863a0015cea392d853216d04c1ed5348839d3d29b083918867502ad Sep 11 14:08:11.070177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.070138 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" event={"ID":"3a5d284a-4668-4379-931d-7aa4c1fd0122","Type":"ContainerStarted","Data":"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262"} Sep 11 14:08:11.070377 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.070208 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:11.070377 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.070234 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" event={"ID":"3a5d284a-4668-4379-931d-7aa4c1fd0122","Type":"ContainerStarted","Data":"b855d645c8804c6c00a1c74608fb9e4bf076fb37310419b8edb5bce2239a8385"} Sep 11 14:08:11.072333 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.072308 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" event={"ID":"b2f810bc-f6f5-437a-a28f-174643e32278","Type":"ContainerStarted","Data":"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e"} Sep 11 14:08:11.072441 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.072353 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" event={"ID":"b2f810bc-f6f5-437a-a28f-174643e32278","Type":"ContainerStarted","Data":"c7e5cecea863a0015cea392d853216d04c1ed5348839d3d29b083918867502ad"} Sep 11 14:08:11.072441 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.072380 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:11.087824 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.087764 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" podStartSLOduration=2.087744591 podStartE2EDuration="2.087744591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:11.084721593 +0000 UTC m=+1158.839036484" watchObservedRunningTime="2026-09-11 14:08:11.087744591 +0000 UTC m=+1158.842059483" Sep 11 14:08:11.104560 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:11.104510 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" podStartSLOduration=2.1044949 podStartE2EDuration="2.1044949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:11.101827654 +0000 UTC m=+1158.856142549" watchObservedRunningTime="2026-09-11 14:08:11.1044949 +0000 UTC m=+1158.858809790" Sep 11 14:08:22.080171 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:22.080093 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:08:22.080533 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:22.080258 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:08:32.962723 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:32.962688 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:36.996852 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:36.996819 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:37.307140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:37.307055 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:37.408170 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:37.408135 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:52.745763 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:52.745731 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:08:52.749385 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:52.749363 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:08:52.869773 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:08:52.869744 2519 scope.go:117] "RemoveContainer" containerID="550c897b0873315a243c5b8746eac3581d56759b42ec4c60eac6dfab14352b95" Sep 11 14:09:17.665654 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.665617 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:17.695652 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.695547 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:09:17.695852 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.695793 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" podUID="3a5d284a-4668-4379-931d-7aa4c1fd0122" containerName="payload-processing" containerID="cri-o://9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262" gracePeriod=30 Sep 11 14:09:17.705893 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.705862 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:09:17.706123 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.706088 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" podUID="b2f810bc-f6f5-437a-a28f-174643e32278" containerName="payload-pre-processing" containerID="cri-o://f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e" gracePeriod=30 Sep 11 14:09:17.994050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.994021 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:09:17.997508 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:17.997480 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:09:18.082686 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.082647 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg\") pod \"3a5d284a-4668-4379-931d-7aa4c1fd0122\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " Sep 11 14:09:18.082904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.082716 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume\") pod \"3a5d284a-4668-4379-931d-7aa4c1fd0122\" (UID: \"3a5d284a-4668-4379-931d-7aa4c1fd0122\") " Sep 11 14:09:18.082904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.082757 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume\") pod \"b2f810bc-f6f5-437a-a28f-174643e32278\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " Sep 11 14:09:18.082904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.082890 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hknxs\" (UniqueName: \"kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs\") pod \"b2f810bc-f6f5-437a-a28f-174643e32278\" (UID: \"b2f810bc-f6f5-437a-a28f-174643e32278\") " Sep 11 14:09:18.083159 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.083124 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a5d284a-4668-4379-931d-7aa4c1fd0122" (UID: "3a5d284a-4668-4379-931d-7aa4c1fd0122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:09:18.083297 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.083158 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2f810bc-f6f5-437a-a28f-174643e32278" (UID: "b2f810bc-f6f5-437a-a28f-174643e32278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:09:18.083355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.083294 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5d284a-4668-4379-931d-7aa4c1fd0122-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:09:18.083355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.083315 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f810bc-f6f5-437a-a28f-174643e32278-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:09:18.084963 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.084939 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs" (OuterVolumeSpecName: "kube-api-access-hknxs") pod "b2f810bc-f6f5-437a-a28f-174643e32278" (UID: "b2f810bc-f6f5-437a-a28f-174643e32278"). InnerVolumeSpecName "kube-api-access-hknxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:18.085066 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.084990 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg" (OuterVolumeSpecName: "kube-api-access-9zvmg") pod "3a5d284a-4668-4379-931d-7aa4c1fd0122" (UID: "3a5d284a-4668-4379-931d-7aa4c1fd0122"). InnerVolumeSpecName "kube-api-access-9zvmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:18.184740 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.184667 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknxs\" (UniqueName: \"kubernetes.io/projected/b2f810bc-f6f5-437a-a28f-174643e32278-kube-api-access-hknxs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:09:18.184740 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.184694 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvmg\" (UniqueName: \"kubernetes.io/projected/3a5d284a-4668-4379-931d-7aa4c1fd0122-kube-api-access-9zvmg\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:09:18.459118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.459025 2519 generic.go:356] "Generic (PLEG): container finished" podID="3a5d284a-4668-4379-931d-7aa4c1fd0122" containerID="9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262" exitCode=0 Sep 11 14:09:18.459118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.459075 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" Sep 11 14:09:18.459118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.459107 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" event={"ID":"3a5d284a-4668-4379-931d-7aa4c1fd0122","Type":"ContainerDied","Data":"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262"} Sep 11 14:09:18.459393 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.459159 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48" event={"ID":"3a5d284a-4668-4379-931d-7aa4c1fd0122","Type":"ContainerDied","Data":"b855d645c8804c6c00a1c74608fb9e4bf076fb37310419b8edb5bce2239a8385"} Sep 11 14:09:18.459393 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.459175 2519 scope.go:117] "RemoveContainer" containerID="9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262" Sep 11 14:09:18.460511 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.460490 2519 generic.go:356] "Generic (PLEG): container finished" podID="b2f810bc-f6f5-437a-a28f-174643e32278" containerID="f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e" exitCode=0 Sep 11 14:09:18.460635 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.460536 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" Sep 11 14:09:18.460635 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.460588 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" event={"ID":"b2f810bc-f6f5-437a-a28f-174643e32278","Type":"ContainerDied","Data":"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e"} Sep 11 14:09:18.460767 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.460634 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr" event={"ID":"b2f810bc-f6f5-437a-a28f-174643e32278","Type":"ContainerDied","Data":"c7e5cecea863a0015cea392d853216d04c1ed5348839d3d29b083918867502ad"} Sep 11 14:09:18.468320 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.468298 2519 scope.go:117] "RemoveContainer" containerID="9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262" Sep 11 14:09:18.468653 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:09:18.468624 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262\": container with ID starting with 9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262 not found: ID does not exist" containerID="9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262" Sep 11 14:09:18.468760 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.468664 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262"} err="failed to get container status \"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262\": rpc error: code = NotFound desc = could not find container \"9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262\": container with ID starting with 9e3a0c5ccb9e33272daa41ed47e0aff4c159067fda492e13f042aa0cb8667262 not found: ID does not exist" Sep 11 14:09:18.468760 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.468690 2519 scope.go:117] "RemoveContainer" containerID="f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e" Sep 11 14:09:18.477404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.477387 2519 scope.go:117] "RemoveContainer" containerID="f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e" Sep 11 14:09:18.477697 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:09:18.477679 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e\": container with ID starting with f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e not found: ID does not exist" containerID="f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e" Sep 11 14:09:18.477741 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.477708 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e"} err="failed to get container status \"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e\": rpc error: code = NotFound desc = could not find container \"f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e\": container with ID starting with f7c92a2c62322e7e82654b309a3b4381c6d496053e58a58a7632fcb9e80bc31e not found: ID does not exist" Sep 11 14:09:18.491126 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.491090 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:09:18.495557 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.495514 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc83333c-645678448b-btwqr"] Sep 11 14:09:18.507412 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.507359 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:09:18.509747 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.509728 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc83333c-b557d67df-hrf48"] Sep 11 14:09:18.691604 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.691547 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5d284a-4668-4379-931d-7aa4c1fd0122" path="/var/lib/kubelet/pods/3a5d284a-4668-4379-931d-7aa4c1fd0122/volumes" Sep 11 14:09:18.691987 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.691931 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f810bc-f6f5-437a-a28f-174643e32278" path="/var/lib/kubelet/pods/b2f810bc-f6f5-437a-a28f-174643e32278/volumes" Sep 11 14:09:18.932900 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:18.932852 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:21.046755 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:21.046718 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:26.125439 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:26.125402 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:45.858950 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:45.858869 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:47.014935 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:47.014904 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:52.703355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:52.703321 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:52.775526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:52.775495 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:52.952437 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:52.952409 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:54.228813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:54.228778 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:54.409161 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:54.409128 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:54.532736 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:54.532662 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:55.143095 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:55.143066 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:55.238842 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:55.238812 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:55.336607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:55.336546 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:56.575746 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:09:56.575714 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:00.939432 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939395 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7543-3293ff"] Sep 11 14:10:00.939907 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939891 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5d284a-4668-4379-931d-7aa4c1fd0122" containerName="payload-processing" Sep 11 14:10:00.939907 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939907 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5d284a-4668-4379-931d-7aa4c1fd0122" containerName="payload-processing" Sep 11 14:10:00.939984 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939919 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f810bc-f6f5-437a-a28f-174643e32278" containerName="payload-pre-processing" Sep 11 14:10:00.939984 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939925 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f810bc-f6f5-437a-a28f-174643e32278" containerName="payload-pre-processing" Sep 11 14:10:00.940058 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.939994 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5d284a-4668-4379-931d-7aa4c1fd0122" containerName="payload-processing" Sep 11 14:10:00.940058 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.940003 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f810bc-f6f5-437a-a28f-174643e32278" containerName="payload-pre-processing" Sep 11 14:10:00.944291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.944273 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:00.946468 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.946443 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2qcb7\"" Sep 11 14:10:00.947757 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:00.947735 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-3293ff"] Sep 11 14:10:01.002682 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.002628 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6\") pod \"test-curl-7543-3293ff\" (UID: \"d645d557-dfd3-4871-925d-ed24ccbfde4d\") " pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:01.103741 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.103705 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6\") pod \"test-curl-7543-3293ff\" (UID: \"d645d557-dfd3-4871-925d-ed24ccbfde4d\") " pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:01.111812 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.111787 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6\") pod \"test-curl-7543-3293ff\" (UID: \"d645d557-dfd3-4871-925d-ed24ccbfde4d\") " pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:01.254536 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.254439 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:01.524447 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.524414 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7543-3293ff"] Sep 11 14:10:01.525801 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:10:01.525776 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd645d557_dfd3_4871_925d_ed24ccbfde4d.slice/crio-51fd1656a8d7de351958f054a280135162c8a7e8f5fcd4cc799575e7e34e78aa WatchSource:0}: Error finding container 51fd1656a8d7de351958f054a280135162c8a7e8f5fcd4cc799575e7e34e78aa: Status 404 returned error can't find the container with id 51fd1656a8d7de351958f054a280135162c8a7e8f5fcd4cc799575e7e34e78aa Sep 11 14:10:01.699039 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.699003 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-3293ff" event={"ID":"d645d557-dfd3-4871-925d-ed24ccbfde4d","Type":"ContainerStarted","Data":"0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48"} Sep 11 14:10:01.699039 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.699044 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7543-3293ff" event={"ID":"d645d557-dfd3-4871-925d-ed24ccbfde4d","Type":"ContainerStarted","Data":"51fd1656a8d7de351958f054a280135162c8a7e8f5fcd4cc799575e7e34e78aa"} Sep 11 14:10:01.715707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.715652 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7543-3293ff" podStartSLOduration=1.7156345979999998 podStartE2EDuration="1.715634598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:01.712997455 +0000 UTC m=+1269.467312347" watchObservedRunningTime="2026-09-11 14:10:01.715634598 +0000 UTC m=+1269.469949490" Sep 11 14:10:01.865541 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:01.865517 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:02.324243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:02.324163 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7543-3293ff"] Sep 11 14:10:02.327608 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:02.327570 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7543-3293ff"] Sep 11 14:10:02.442285 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:02.442256 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:02.544221 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:02.544188 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:02.703948 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:10:02.703919 2519 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7543-3293ff" container="test-curl-7543-3293ff" Sep 11 14:10:03.102651 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:03.102619 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:03.709373 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:03.709335 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7543-3293ff" podUID="d645d557-dfd3-4871-925d-ed24ccbfde4d" containerName="test-curl-7543-3293ff" containerID="cri-o://0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48" gracePeriod=2 Sep 11 14:10:03.733589 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:03.733543 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:04.772877 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:04.772846 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:05.891213 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:05.891189 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:05.957534 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:05.957502 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6\") pod \"d645d557-dfd3-4871-925d-ed24ccbfde4d\" (UID: \"d645d557-dfd3-4871-925d-ed24ccbfde4d\") " Sep 11 14:10:05.959519 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:05.959495 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6" (OuterVolumeSpecName: "kube-api-access-mj4q6") pod "d645d557-dfd3-4871-925d-ed24ccbfde4d" (UID: "d645d557-dfd3-4871-925d-ed24ccbfde4d"). InnerVolumeSpecName "kube-api-access-mj4q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:10:06.059304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.059265 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4q6\" (UniqueName: \"kubernetes.io/projected/d645d557-dfd3-4871-925d-ed24ccbfde4d-kube-api-access-mj4q6\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:10:06.691515 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.691484 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d645d557-dfd3-4871-925d-ed24ccbfde4d" path="/var/lib/kubelet/pods/d645d557-dfd3-4871-925d-ed24ccbfde4d/volumes" Sep 11 14:10:06.725565 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.725533 2519 generic.go:356] "Generic (PLEG): container finished" podID="d645d557-dfd3-4871-925d-ed24ccbfde4d" containerID="0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48" exitCode=137 Sep 11 14:10:06.725565 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.725564 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7543-3293ff" Sep 11 14:10:06.725742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.725601 2519 scope.go:117] "RemoveContainer" containerID="0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48" Sep 11 14:10:06.738144 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.738128 2519 scope.go:117] "RemoveContainer" containerID="0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48" Sep 11 14:10:06.738450 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:10:06.738426 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48\": container with ID starting with 0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48 not found: ID does not exist" containerID="0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48" Sep 11 14:10:06.738509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:06.738455 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48"} err="failed to get container status \"0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48\": rpc error: code = NotFound desc = could not find container \"0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48\": container with ID starting with 0b230ad862bb8609f523a6753f4f1594b623c284e34bf77434b6cba70191ec48 not found: ID does not exist" Sep 11 14:10:09.436625 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:09.436571 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:09.534174 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:09.534143 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:09.631250 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:09.631220 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:20.163362 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:20.163325 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:20.401906 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:20.401868 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:20.992292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:20.992259 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:41.348225 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:41.348191 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.064050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:47.064010 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.163545 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:47.163502 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.423903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:47.423871 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.747952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:47.747855 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.860228 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:47.860185 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:51.303815 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:51.303776 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:51.409820 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:51.409788 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:52.565725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:52.565692 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:52.668402 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:52.668348 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:53.193823 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.193727 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:53.294368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.294330 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:53.318015 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.317983 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:10:53.318480 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.318465 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d645d557-dfd3-4871-925d-ed24ccbfde4d" containerName="test-curl-7543-3293ff" Sep 11 14:10:53.318533 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.318481 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="d645d557-dfd3-4871-925d-ed24ccbfde4d" containerName="test-curl-7543-3293ff" Sep 11 14:10:53.318603 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.318555 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="d645d557-dfd3-4871-925d-ed24ccbfde4d" containerName="test-curl-7543-3293ff" Sep 11 14:10:53.323135 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.323111 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.325101 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.325078 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-427751\"" Sep 11 14:10:53.334517 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.334499 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:10:53.337105 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.337082 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:10:53.340944 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.340923 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.343413 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.343393 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-427751-dockercfg-mjtll\"" Sep 11 14:10:53.353204 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.353182 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:10:53.424865 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.424833 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6pq\" (UniqueName: \"kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.425037 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.424894 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.525526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.525442 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6pq\" (UniqueName: \"kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.525710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.525524 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.525710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.525557 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szxv\" (UniqueName: \"kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.525710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.525652 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.526300 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.526283 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.533321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.533292 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6pq\" (UniqueName: \"kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq\") pod \"payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.626600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.626553 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.626965 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.626607 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szxv\" (UniqueName: \"kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.627318 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.627298 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.634918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.634896 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:53.635247 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.635229 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szxv\" (UniqueName: \"kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv\") pod \"payload-processing-e2e-api-a-427751-58b56c8447-c5422\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.654321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.654294 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:53.907394 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.907366 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:10:53.909265 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:10:53.909238 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb696ea_2779_4d2b_947f_a3d3d356bdb9.slice/crio-5294524c6cc1333f3af52613bec80da9ef24ea3d36c4f2268629c4c0add273d0 WatchSource:0}: Error finding container 5294524c6cc1333f3af52613bec80da9ef24ea3d36c4f2268629c4c0add273d0: Status 404 returned error can't find the container with id 5294524c6cc1333f3af52613bec80da9ef24ea3d36c4f2268629c4c0add273d0 Sep 11 14:10:53.940454 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.940431 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:10:53.945751 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:10:53.945722 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddad024_a3f7_4780_bc78_47eaa87c6ad4.slice/crio-b9ee662e7f13fc4f4a4abae2873c0f4cacbdf3ee46fbad39adab4e5863b62dd0 WatchSource:0}: Error finding container b9ee662e7f13fc4f4a4abae2873c0f4cacbdf3ee46fbad39adab4e5863b62dd0: Status 404 returned error can't find the container with id b9ee662e7f13fc4f4a4abae2873c0f4cacbdf3ee46fbad39adab4e5863b62dd0 Sep 11 14:10:53.985419 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.985392 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" event={"ID":"3ddad024-a3f7-4780-bc78-47eaa87c6ad4","Type":"ContainerStarted","Data":"b9ee662e7f13fc4f4a4abae2873c0f4cacbdf3ee46fbad39adab4e5863b62dd0"} Sep 11 14:10:53.987056 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.987032 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" event={"ID":"ccb696ea-2779-4d2b-947f-a3d3d356bdb9","Type":"ContainerStarted","Data":"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a"} Sep 11 14:10:53.987155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.987083 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" event={"ID":"ccb696ea-2779-4d2b-947f-a3d3d356bdb9","Type":"ContainerStarted","Data":"5294524c6cc1333f3af52613bec80da9ef24ea3d36c4f2268629c4c0add273d0"} Sep 11 14:10:53.987155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:53.987115 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:10:54.004467 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:54.004419 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" podStartSLOduration=1.004403962 podStartE2EDuration="1.004403962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:54.001380922 +0000 UTC m=+1321.755695814" watchObservedRunningTime="2026-09-11 14:10:54.004403962 +0000 UTC m=+1321.758718852" Sep 11 14:10:54.994637 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:54.994602 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" event={"ID":"3ddad024-a3f7-4780-bc78-47eaa87c6ad4","Type":"ContainerStarted","Data":"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752"} Sep 11 14:10:54.995131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:54.994917 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:10:55.012401 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:10:55.012352 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" podStartSLOduration=2.0123371 podStartE2EDuration="2.0123371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:55.010997886 +0000 UTC m=+1322.765312779" watchObservedRunningTime="2026-09-11 14:10:55.0123371 +0000 UTC m=+1322.766651990" Sep 11 14:11:02.494819 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:02.494786 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:04.996431 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:04.996396 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:11:06.001869 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:06.001834 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:11:16.301251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:16.301181 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:16.846842 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:16.846803 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:17.159874 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:17.159789 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:17.402218 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:17.402183 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:17.508141 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:17.508064 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:20.270019 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:20.269983 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:20.363358 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:20.363316 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:21.490848 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:21.490819 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:21.591612 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:21.591565 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:21.657840 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:21.657810 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:22.003446 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.003403 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:22.102260 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.102229 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:22.120781 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.120747 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:11:22.124905 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.124879 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.127685 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.127655 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1f2b49-dockercfg-mwcxj\"" Sep 11 14:11:22.127925 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.127909 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1f2b49\"" Sep 11 14:11:22.134352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.134331 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:11:22.214357 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.214315 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlb9\" (UniqueName: \"kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.214357 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.214357 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.315598 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.315498 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlb9\" (UniqueName: \"kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.315598 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.315542 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.316234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.316212 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.323194 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.323170 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlb9\" (UniqueName: \"kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9\") pod \"payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.438399 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.438346 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:22.714897 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:22.714863 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:11:22.718156 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:11:22.718128 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82334a1f_d638_4ea6_af62_1303bdec2caf.slice/crio-aabbd1e4374f7eb2d796f4cecd1b599b0c0d26d549089557f70b0e0dc38f6186 WatchSource:0}: Error finding container aabbd1e4374f7eb2d796f4cecd1b599b0c0d26d549089557f70b0e0dc38f6186: Status 404 returned error can't find the container with id aabbd1e4374f7eb2d796f4cecd1b599b0c0d26d549089557f70b0e0dc38f6186 Sep 11 14:11:23.157586 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:23.157536 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" event={"ID":"82334a1f-d638-4ea6-af62-1303bdec2caf","Type":"ContainerStarted","Data":"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2"} Sep 11 14:11:23.157928 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:23.157606 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" event={"ID":"82334a1f-d638-4ea6-af62-1303bdec2caf","Type":"ContainerStarted","Data":"aabbd1e4374f7eb2d796f4cecd1b599b0c0d26d549089557f70b0e0dc38f6186"} Sep 11 14:11:23.157928 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:23.157729 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:23.176518 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:23.176468 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" podStartSLOduration=1.176451242 podStartE2EDuration="1.176451242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:23.173831768 +0000 UTC m=+1350.928146660" watchObservedRunningTime="2026-09-11 14:11:23.176451242 +0000 UTC m=+1350.930766133" Sep 11 14:11:23.978151 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:23.978103 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:34.166636 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:34.166570 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:11:42.419985 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:42.419952 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:45.219117 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:45.219086 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:48.144235 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:48.144205 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:48.242165 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:48.242124 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.186453 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:49.186408 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:52.654410 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:52.654343 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:52.940172 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:52.940083 2519 scope.go:117] "RemoveContainer" containerID="b47c82e345e2278b4558e073b357d93e3ae2c2b72ac9babefef64ff72c40adbe" Sep 11 14:11:52.949148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:52.949128 2519 scope.go:117] "RemoveContainer" containerID="14493c87102823888f64fbacac453af6a2417b9ac008a375366c92921b13a2b4" Sep 11 14:11:53.076267 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:53.076217 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:58.190243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:11:58.190205 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.823453 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.823398 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.853316 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.853246 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:12:03.853589 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.853524 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" podUID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" containerName="payload-processing" containerID="cri-o://128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752" gracePeriod=30 Sep 11 14:12:03.862413 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.862386 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:12:03.862735 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.862707 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" podUID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" containerName="payload-pre-processing" containerID="cri-o://cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a" gracePeriod=30 Sep 11 14:12:03.907952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.907899 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.998277 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:03.998239 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:04.068156 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.068120 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:04.125183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.125157 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:12:04.155509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.155479 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:12:04.247397 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.247360 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2szxv\" (UniqueName: \"kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv\") pod \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " Sep 11 14:12:04.247644 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.247485 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume\") pod \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\" (UID: \"3ddad024-a3f7-4780-bc78-47eaa87c6ad4\") " Sep 11 14:12:04.247891 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.247865 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ddad024-a3f7-4780-bc78-47eaa87c6ad4" (UID: "3ddad024-a3f7-4780-bc78-47eaa87c6ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:04.249558 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.249531 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv" (OuterVolumeSpecName: "kube-api-access-2szxv") pod "3ddad024-a3f7-4780-bc78-47eaa87c6ad4" (UID: "3ddad024-a3f7-4780-bc78-47eaa87c6ad4"). InnerVolumeSpecName "kube-api-access-2szxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:04.348898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.348867 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv6pq\" (UniqueName: \"kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq\") pod \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " Sep 11 14:12:04.349075 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.348962 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume\") pod \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\" (UID: \"ccb696ea-2779-4d2b-947f-a3d3d356bdb9\") " Sep 11 14:12:04.349221 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.349208 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:04.349262 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.349226 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szxv\" (UniqueName: \"kubernetes.io/projected/3ddad024-a3f7-4780-bc78-47eaa87c6ad4-kube-api-access-2szxv\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:04.349335 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.349310 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccb696ea-2779-4d2b-947f-a3d3d356bdb9" (UID: "ccb696ea-2779-4d2b-947f-a3d3d356bdb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:04.351091 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.351065 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq" (OuterVolumeSpecName: "kube-api-access-nv6pq") pod "ccb696ea-2779-4d2b-947f-a3d3d356bdb9" (UID: "ccb696ea-2779-4d2b-947f-a3d3d356bdb9"). InnerVolumeSpecName "kube-api-access-nv6pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:04.392923 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.392890 2519 generic.go:356] "Generic (PLEG): container finished" podID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" containerID="128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752" exitCode=0 Sep 11 14:12:04.393102 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.392934 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" Sep 11 14:12:04.393102 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.392979 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" event={"ID":"3ddad024-a3f7-4780-bc78-47eaa87c6ad4","Type":"ContainerDied","Data":"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752"} Sep 11 14:12:04.393102 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.393057 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422" event={"ID":"3ddad024-a3f7-4780-bc78-47eaa87c6ad4","Type":"ContainerDied","Data":"b9ee662e7f13fc4f4a4abae2873c0f4cacbdf3ee46fbad39adab4e5863b62dd0"} Sep 11 14:12:04.393285 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.393097 2519 scope.go:117] "RemoveContainer" containerID="128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752" Sep 11 14:12:04.394396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.394300 2519 generic.go:356] "Generic (PLEG): container finished" podID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" containerID="cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a" exitCode=0 Sep 11 14:12:04.394396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.394355 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" Sep 11 14:12:04.394505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.394390 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" event={"ID":"ccb696ea-2779-4d2b-947f-a3d3d356bdb9","Type":"ContainerDied","Data":"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a"} Sep 11 14:12:04.394505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.394434 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb" event={"ID":"ccb696ea-2779-4d2b-947f-a3d3d356bdb9","Type":"ContainerDied","Data":"5294524c6cc1333f3af52613bec80da9ef24ea3d36c4f2268629c4c0add273d0"} Sep 11 14:12:04.403535 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.403505 2519 scope.go:117] "RemoveContainer" containerID="128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752" Sep 11 14:12:04.403870 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:12:04.403825 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752\": container with ID starting with 128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752 not found: ID does not exist" containerID="128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752" Sep 11 14:12:04.403952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.403865 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752"} err="failed to get container status \"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752\": rpc error: code = NotFound desc = could not find container \"128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752\": container with ID starting with 128ea414ae495716dd283e325328ff52be91219c23e18a014166eb898c28a752 not found: ID does not exist" Sep 11 14:12:04.403952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.403882 2519 scope.go:117] "RemoveContainer" containerID="cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a" Sep 11 14:12:04.412317 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.412301 2519 scope.go:117] "RemoveContainer" containerID="cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a" Sep 11 14:12:04.412565 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:12:04.412549 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a\": container with ID starting with cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a not found: ID does not exist" containerID="cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a" Sep 11 14:12:04.412756 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.412585 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a"} err="failed to get container status \"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a\": rpc error: code = NotFound desc = could not find container \"cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a\": container with ID starting with cb10393cee69b907bb1333be3b731ef1852e16745f559797d5e2062b3945f42a not found: ID does not exist" Sep 11 14:12:04.418885 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.418863 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:12:04.422895 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.422872 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-427751-f6f65fc46-m72zb"] Sep 11 14:12:04.432164 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.432139 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:12:04.439336 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.439312 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-427751-58b56c8447-c5422"] Sep 11 14:12:04.449945 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.449926 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6pq\" (UniqueName: \"kubernetes.io/projected/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-kube-api-access-nv6pq\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:04.450033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.449947 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb696ea-2779-4d2b-947f-a3d3d356bdb9-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:04.454736 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.454713 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:04.692609 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.692519 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" path="/var/lib/kubelet/pods/3ddad024-a3f7-4780-bc78-47eaa87c6ad4/volumes" Sep 11 14:12:04.692974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:04.692959 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" path="/var/lib/kubelet/pods/ccb696ea-2779-4d2b-947f-a3d3d356bdb9/volumes" Sep 11 14:12:10.528848 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:10.528812 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.609694 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:10.609662 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.689012 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:10.688976 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.806591 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:10.806480 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:11.416220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:11.416186 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:22.766916 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:22.766866 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:26.781414 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:26.781382 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:26.822234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:26.822195 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:12:26.822826 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:26.822774 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" podUID="82334a1f-d638-4ea6-af62-1303bdec2caf" containerName="payload-pre-processing" containerID="cri-o://996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2" gracePeriod=30 Sep 11 14:12:26.882180 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:26.882150 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:26.982481 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:26.982447 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:27.083161 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.083132 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:27.084238 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.084216 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:12:27.179646 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.179603 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnlb9\" (UniqueName: \"kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9\") pod \"82334a1f-d638-4ea6-af62-1303bdec2caf\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " Sep 11 14:12:27.179827 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.179725 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume\") pod \"82334a1f-d638-4ea6-af62-1303bdec2caf\" (UID: \"82334a1f-d638-4ea6-af62-1303bdec2caf\") " Sep 11 14:12:27.180096 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.180070 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82334a1f-d638-4ea6-af62-1303bdec2caf" (UID: "82334a1f-d638-4ea6-af62-1303bdec2caf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:27.181844 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.181818 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9" (OuterVolumeSpecName: "kube-api-access-lnlb9") pod "82334a1f-d638-4ea6-af62-1303bdec2caf" (UID: "82334a1f-d638-4ea6-af62-1303bdec2caf"). InnerVolumeSpecName "kube-api-access-lnlb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:27.280991 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.280957 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82334a1f-d638-4ea6-af62-1303bdec2caf-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:27.280991 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.280986 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlb9\" (UniqueName: \"kubernetes.io/projected/82334a1f-d638-4ea6-af62-1303bdec2caf-kube-api-access-lnlb9\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:27.415159 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.415126 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:27.536262 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.536222 2519 generic.go:356] "Generic (PLEG): container finished" podID="82334a1f-d638-4ea6-af62-1303bdec2caf" containerID="996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2" exitCode=0 Sep 11 14:12:27.536262 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.536267 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" Sep 11 14:12:27.536500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.536296 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" event={"ID":"82334a1f-d638-4ea6-af62-1303bdec2caf","Type":"ContainerDied","Data":"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2"} Sep 11 14:12:27.536500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.536369 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx" event={"ID":"82334a1f-d638-4ea6-af62-1303bdec2caf","Type":"ContainerDied","Data":"aabbd1e4374f7eb2d796f4cecd1b599b0c0d26d549089557f70b0e0dc38f6186"} Sep 11 14:12:27.536500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.536390 2519 scope.go:117] "RemoveContainer" containerID="996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2" Sep 11 14:12:27.549118 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.549095 2519 scope.go:117] "RemoveContainer" containerID="996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2" Sep 11 14:12:27.549432 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:12:27.549414 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2\": container with ID starting with 996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2 not found: ID does not exist" containerID="996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2" Sep 11 14:12:27.549493 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.549440 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2"} err="failed to get container status \"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2\": rpc error: code = NotFound desc = could not find container \"996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2\": container with ID starting with 996c8613f5a03fb9eeab0f743c53b3722a9ac8c507a353fe79951ef5f44e1ba2 not found: ID does not exist" Sep 11 14:12:27.558137 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.558108 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:12:27.561730 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:27.561707 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1f2b49-749dc94bfd-cz8sx"] Sep 11 14:12:28.691766 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:28.691733 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82334a1f-d638-4ea6-af62-1303bdec2caf" path="/var/lib/kubelet/pods/82334a1f-d638-4ea6-af62-1303bdec2caf/volumes" Sep 11 14:12:29.071893 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:29.071804 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:34.555174 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:34.555138 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:34.670003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:34.669975 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:34.893764 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:34.893733 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:35.234947 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.234858 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k"] Sep 11 14:12:35.235327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235311 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" containerName="payload-pre-processing" Sep 11 14:12:35.235327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235325 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" containerName="payload-pre-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235335 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" containerName="payload-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235341 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" containerName="payload-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235358 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82334a1f-d638-4ea6-af62-1303bdec2caf" containerName="payload-pre-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235364 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="82334a1f-d638-4ea6-af62-1303bdec2caf" containerName="payload-pre-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235440 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb696ea-2779-4d2b-947f-a3d3d356bdb9" containerName="payload-pre-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235449 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ddad024-a3f7-4780-bc78-47eaa87c6ad4" containerName="payload-processing" Sep 11 14:12:35.235514 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.235457 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="82334a1f-d638-4ea6-af62-1303bdec2caf" containerName="payload-pre-processing" Sep 11 14:12:35.240005 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.239985 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:35.242445 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.242418 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:12:35.246380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.246349 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k"] Sep 11 14:12:35.365033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.364997 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k\" (UID: \"74766326-e685-415c-9e28-37a49ecbe131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:35.466238 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.466201 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k\" (UID: \"74766326-e685-415c-9e28-37a49ecbe131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:35.466931 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.466911 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k\" (UID: \"74766326-e685-415c-9e28-37a49ecbe131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:35.553091 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.553004 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:35.825813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:35.825780 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k"] Sep 11 14:12:35.827195 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:12:35.827166 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74766326_e685_415c_9e28_37a49ecbe131.slice/crio-c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e WatchSource:0}: Error finding container c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e: Status 404 returned error can't find the container with id c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e Sep 11 14:12:36.590542 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:36.590506 2519 generic.go:356] "Generic (PLEG): container finished" podID="74766326-e685-415c-9e28-37a49ecbe131" containerID="6cf853d4443f1811bb0713a48885a84bee91adf74b3a3a8c51080a811defd7f7" exitCode=0 Sep 11 14:12:36.590759 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:36.590618 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" event={"ID":"74766326-e685-415c-9e28-37a49ecbe131","Type":"ContainerDied","Data":"6cf853d4443f1811bb0713a48885a84bee91adf74b3a3a8c51080a811defd7f7"} Sep 11 14:12:36.590759 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:36.590678 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" event={"ID":"74766326-e685-415c-9e28-37a49ecbe131","Type":"ContainerStarted","Data":"c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e"} Sep 11 14:12:37.732044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:37.732019 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:37.889750 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:37.889663 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca\") pod \"74766326-e685-415c-9e28-37a49ecbe131\" (UID: \"74766326-e685-415c-9e28-37a49ecbe131\") " Sep 11 14:12:37.890053 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:37.890026 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74766326-e685-415c-9e28-37a49ecbe131" (UID: "74766326-e685-415c-9e28-37a49ecbe131"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:37.990783 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:37.990748 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74766326-e685-415c-9e28-37a49ecbe131-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:38.602930 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:38.602901 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" Sep 11 14:12:38.603081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:38.602927 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k" event={"ID":"74766326-e685-415c-9e28-37a49ecbe131","Type":"ContainerDied","Data":"c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e"} Sep 11 14:12:38.603081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:38.602997 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bdbdc5267102ed18ad9e9a0017fd32731e1b4fd0c3e3b670332f8a5558e45e" Sep 11 14:12:45.862599 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.862510 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:45.896901 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.896316 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:12:45.896901 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.896642 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" podUID="41119af7-58d4-4203-8438-933fccd87d6f" containerName="payload-processing" containerID="cri-o://29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919" gracePeriod=30 Sep 11 14:12:45.901520 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.901494 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:12:45.901732 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.901691 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" podUID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" containerName="payload-pre-processing" containerID="cri-o://dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1" gracePeriod=30 Sep 11 14:12:45.936198 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:45.936171 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.002912 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.002876 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.080717 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.080691 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k"] Sep 11 14:12:46.084287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.084251 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b9470-rsh8k"] Sep 11 14:12:46.084815 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.084793 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.149163 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.149116 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.151630 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.151609 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:12:46.162487 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.162463 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume\") pod \"41119af7-58d4-4203-8438-933fccd87d6f\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " Sep 11 14:12:46.162651 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.162534 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9mc\" (UniqueName: \"kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc\") pod \"41119af7-58d4-4203-8438-933fccd87d6f\" (UID: \"41119af7-58d4-4203-8438-933fccd87d6f\") " Sep 11 14:12:46.162932 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.162911 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41119af7-58d4-4203-8438-933fccd87d6f" (UID: "41119af7-58d4-4203-8438-933fccd87d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:46.164908 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.164886 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc" (OuterVolumeSpecName: "kube-api-access-qx9mc") pod "41119af7-58d4-4203-8438-933fccd87d6f" (UID: "41119af7-58d4-4203-8438-933fccd87d6f"). InnerVolumeSpecName "kube-api-access-qx9mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:46.184214 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.184185 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:12:46.263450 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.263418 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbjbm\" (UniqueName: \"kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm\") pod \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " Sep 11 14:12:46.263639 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.263481 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume\") pod \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\" (UID: \"28be26a6-72be-410a-a933-cfa5b2aa1c4d\") " Sep 11 14:12:46.263880 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.263858 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28be26a6-72be-410a-a933-cfa5b2aa1c4d" (UID: "28be26a6-72be-410a-a933-cfa5b2aa1c4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:46.263926 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.263866 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41119af7-58d4-4203-8438-933fccd87d6f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:46.263926 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.263906 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9mc\" (UniqueName: \"kubernetes.io/projected/41119af7-58d4-4203-8438-933fccd87d6f-kube-api-access-qx9mc\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:46.265638 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.265618 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm" (OuterVolumeSpecName: "kube-api-access-sbjbm") pod "28be26a6-72be-410a-a933-cfa5b2aa1c4d" (UID: "28be26a6-72be-410a-a933-cfa5b2aa1c4d"). InnerVolumeSpecName "kube-api-access-sbjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:46.364570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.364530 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjbm\" (UniqueName: \"kubernetes.io/projected/28be26a6-72be-410a-a933-cfa5b2aa1c4d-kube-api-access-sbjbm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:46.364570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.364563 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28be26a6-72be-410a-a933-cfa5b2aa1c4d-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:46.440899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.440810 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.648772 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.648729 2519 generic.go:356] "Generic (PLEG): container finished" podID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" containerID="dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1" exitCode=0 Sep 11 14:12:46.648938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.648780 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" Sep 11 14:12:46.648938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.648815 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" event={"ID":"28be26a6-72be-410a-a933-cfa5b2aa1c4d","Type":"ContainerDied","Data":"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1"} Sep 11 14:12:46.648938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.648871 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567" event={"ID":"28be26a6-72be-410a-a933-cfa5b2aa1c4d","Type":"ContainerDied","Data":"84599eb7a2ce0605c44058c3c9647406b97f9b6c23bdd2dfe0b23670230e830a"} Sep 11 14:12:46.648938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.648896 2519 scope.go:117] "RemoveContainer" containerID="dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1" Sep 11 14:12:46.650056 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.650034 2519 generic.go:356] "Generic (PLEG): container finished" podID="41119af7-58d4-4203-8438-933fccd87d6f" containerID="29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919" exitCode=0 Sep 11 14:12:46.650133 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.650074 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" Sep 11 14:12:46.650177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.650127 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" event={"ID":"41119af7-58d4-4203-8438-933fccd87d6f","Type":"ContainerDied","Data":"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919"} Sep 11 14:12:46.650177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.650169 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8" event={"ID":"41119af7-58d4-4203-8438-933fccd87d6f","Type":"ContainerDied","Data":"049c5313344e28b99a4be0525a6b462da35bbcbbc8898d858e67080f5af21966"} Sep 11 14:12:46.658075 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.658055 2519 scope.go:117] "RemoveContainer" containerID="dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1" Sep 11 14:12:46.658346 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:12:46.658327 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1\": container with ID starting with dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1 not found: ID does not exist" containerID="dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1" Sep 11 14:12:46.658431 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.658353 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1"} err="failed to get container status \"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1\": rpc error: code = NotFound desc = could not find container \"dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1\": container with ID starting with dabccb36ea8763b4445993a437447ffe6d3aeedab7bd991768ec53fdbb12c3a1 not found: ID does not exist" Sep 11 14:12:46.658431 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.658368 2519 scope.go:117] "RemoveContainer" containerID="29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919" Sep 11 14:12:46.666630 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.666609 2519 scope.go:117] "RemoveContainer" containerID="29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919" Sep 11 14:12:46.666883 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:12:46.666866 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919\": container with ID starting with 29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919 not found: ID does not exist" containerID="29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919" Sep 11 14:12:46.666935 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.666889 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919"} err="failed to get container status \"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919\": rpc error: code = NotFound desc = could not find container \"29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919\": container with ID starting with 29b88d2f7ba901876539af2334539e2622b75f29267ef6b028a19efe3c850919 not found: ID does not exist" Sep 11 14:12:46.676131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.676112 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:12:46.680095 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.680074 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b9470-79b499ccbb-k9567"] Sep 11 14:12:46.693456 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.693391 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" path="/var/lib/kubelet/pods/28be26a6-72be-410a-a933-cfa5b2aa1c4d/volumes" Sep 11 14:12:46.693786 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.693772 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74766326-e685-415c-9e28-37a49ecbe131" path="/var/lib/kubelet/pods/74766326-e685-415c-9e28-37a49ecbe131/volumes" Sep 11 14:12:46.694022 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.694009 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:12:46.694213 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.694194 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6b9470-7894c85b69-55xq8"] Sep 11 14:12:46.733781 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.733752 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.795722 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:46.795695 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:48.691164 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:48.691131 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41119af7-58d4-4203-8438-933fccd87d6f" path="/var/lib/kubelet/pods/41119af7-58d4-4203-8438-933fccd87d6f/volumes" Sep 11 14:12:50.304299 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:50.304266 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:51.547011 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:51.546981 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:52.480131 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:52.480097 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:52.583690 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:52.583655 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:52.737777 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:52.737690 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:53.089526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.089485 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b"] Sep 11 14:12:53.090197 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090168 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" containerName="payload-pre-processing" Sep 11 14:12:53.090197 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090197 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" containerName="payload-pre-processing" Sep 11 14:12:53.090366 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090224 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74766326-e685-415c-9e28-37a49ecbe131" containerName="revoke-keys" Sep 11 14:12:53.090366 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090235 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="74766326-e685-415c-9e28-37a49ecbe131" containerName="revoke-keys" Sep 11 14:12:53.090366 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090266 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41119af7-58d4-4203-8438-933fccd87d6f" containerName="payload-processing" Sep 11 14:12:53.090366 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090274 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="41119af7-58d4-4203-8438-933fccd87d6f" containerName="payload-processing" Sep 11 14:12:53.090533 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090400 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="41119af7-58d4-4203-8438-933fccd87d6f" containerName="payload-processing" Sep 11 14:12:53.090533 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090414 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="28be26a6-72be-410a-a933-cfa5b2aa1c4d" containerName="payload-pre-processing" Sep 11 14:12:53.090533 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.090428 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="74766326-e685-415c-9e28-37a49ecbe131" containerName="revoke-keys" Sep 11 14:12:53.095881 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.095856 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:53.098470 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.098360 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:12:53.099018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.098849 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b"] Sep 11 14:12:53.131125 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.131099 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b\" (UID: \"d1362cd8-f1b9-41ea-865b-3c9fd197c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:53.232033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.231992 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b\" (UID: \"d1362cd8-f1b9-41ea-865b-3c9fd197c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:53.232661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.232642 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b\" (UID: \"d1362cd8-f1b9-41ea-865b-3c9fd197c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:53.411593 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.411550 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:53.684420 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.684395 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b"] Sep 11 14:12:53.686774 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:12:53.686748 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1362cd8_f1b9_41ea_865b_3c9fd197c081.slice/crio-55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae WatchSource:0}: Error finding container 55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae: Status 404 returned error can't find the container with id 55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae Sep 11 14:12:53.700287 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:53.700263 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" event={"ID":"d1362cd8-f1b9-41ea-865b-3c9fd197c081","Type":"ContainerStarted","Data":"55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae"} Sep 11 14:12:54.706081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:54.706051 2519 generic.go:356] "Generic (PLEG): container finished" podID="d1362cd8-f1b9-41ea-865b-3c9fd197c081" containerID="e8a8b6782af75b9865178869faf831973a0c81cd0ab37bc5f0598c9b7ddef93c" exitCode=0 Sep 11 14:12:54.706483 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:54.706134 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" event={"ID":"d1362cd8-f1b9-41ea-865b-3c9fd197c081","Type":"ContainerDied","Data":"e8a8b6782af75b9865178869faf831973a0c81cd0ab37bc5f0598c9b7ddef93c"} Sep 11 14:12:55.409903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:55.409859 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:55.852244 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:55.852216 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:55.964461 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:55.964367 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca\") pod \"d1362cd8-f1b9-41ea-865b-3c9fd197c081\" (UID: \"d1362cd8-f1b9-41ea-865b-3c9fd197c081\") " Sep 11 14:12:55.964798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:55.964777 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1362cd8-f1b9-41ea-865b-3c9fd197c081" (UID: "d1362cd8-f1b9-41ea-865b-3c9fd197c081"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:56.065321 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:56.065287 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1362cd8-f1b9-41ea-865b-3c9fd197c081-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:12:56.718782 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:56.718751 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" Sep 11 14:12:56.718930 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:56.718815 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b" event={"ID":"d1362cd8-f1b9-41ea-865b-3c9fd197c081","Type":"ContainerDied","Data":"55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae"} Sep 11 14:12:56.718930 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:12:56.718862 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55375dba87070c8028e6cb9017792bbf2883eb2746b71ca4c5ef45a4186ed4ae" Sep 11 14:13:03.253450 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:03.253408 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.011340 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.011309 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.042670 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.042627 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:13:04.042921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.042892 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" podUID="619dcec7-56c1-45a8-9bed-b89435d1d652" containerName="payload-processing" containerID="cri-o://3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec" gracePeriod=30 Sep 11 14:13:04.049341 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.049310 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:13:04.049558 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.049534 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" podUID="f4561b9a-3861-443f-9fac-04c21591c3ed" containerName="payload-pre-processing" containerID="cri-o://1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27" gracePeriod=30 Sep 11 14:13:04.090994 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.090968 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.153092 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.153067 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.229062 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.229032 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b"] Sep 11 14:13:04.230318 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.230288 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.234898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.234870 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01faa9-2dz2b"] Sep 11 14:13:04.294591 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.294542 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.298098 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.298077 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:13:04.333435 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.333409 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:13:04.449981 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.449949 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68xbd\" (UniqueName: \"kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd\") pod \"619dcec7-56c1-45a8-9bed-b89435d1d652\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " Sep 11 14:13:04.450177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.449992 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6znzx\" (UniqueName: \"kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx\") pod \"f4561b9a-3861-443f-9fac-04c21591c3ed\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " Sep 11 14:13:04.450177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.450030 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume\") pod \"f4561b9a-3861-443f-9fac-04c21591c3ed\" (UID: \"f4561b9a-3861-443f-9fac-04c21591c3ed\") " Sep 11 14:13:04.450177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.450059 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume\") pod \"619dcec7-56c1-45a8-9bed-b89435d1d652\" (UID: \"619dcec7-56c1-45a8-9bed-b89435d1d652\") " Sep 11 14:13:04.450452 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.450420 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4561b9a-3861-443f-9fac-04c21591c3ed" (UID: "f4561b9a-3861-443f-9fac-04c21591c3ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:13:04.450452 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.450442 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "619dcec7-56c1-45a8-9bed-b89435d1d652" (UID: "619dcec7-56c1-45a8-9bed-b89435d1d652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:13:04.452179 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.452157 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd" (OuterVolumeSpecName: "kube-api-access-68xbd") pod "619dcec7-56c1-45a8-9bed-b89435d1d652" (UID: "619dcec7-56c1-45a8-9bed-b89435d1d652"). InnerVolumeSpecName "kube-api-access-68xbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:13:04.452243 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.452203 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx" (OuterVolumeSpecName: "kube-api-access-6znzx") pod "f4561b9a-3861-443f-9fac-04c21591c3ed" (UID: "f4561b9a-3861-443f-9fac-04c21591c3ed"). InnerVolumeSpecName "kube-api-access-6znzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:13:04.551251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.551178 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619dcec7-56c1-45a8-9bed-b89435d1d652-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:13:04.551251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.551205 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68xbd\" (UniqueName: \"kubernetes.io/projected/619dcec7-56c1-45a8-9bed-b89435d1d652-kube-api-access-68xbd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:13:04.551251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.551214 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znzx\" (UniqueName: \"kubernetes.io/projected/f4561b9a-3861-443f-9fac-04c21591c3ed-kube-api-access-6znzx\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:13:04.551251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.551224 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4561b9a-3861-443f-9fac-04c21591c3ed-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:13:04.603274 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.603236 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:04.691830 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.691792 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1362cd8-f1b9-41ea-865b-3c9fd197c081" path="/var/lib/kubelet/pods/d1362cd8-f1b9-41ea-865b-3c9fd197c081/volumes" Sep 11 14:13:04.762616 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.762560 2519 generic.go:356] "Generic (PLEG): container finished" podID="f4561b9a-3861-443f-9fac-04c21591c3ed" containerID="1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27" exitCode=0 Sep 11 14:13:04.762616 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.762608 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" Sep 11 14:13:04.762847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.762646 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" event={"ID":"f4561b9a-3861-443f-9fac-04c21591c3ed","Type":"ContainerDied","Data":"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27"} Sep 11 14:13:04.762847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.762701 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg" event={"ID":"f4561b9a-3861-443f-9fac-04c21591c3ed","Type":"ContainerDied","Data":"65dc9ea58ee7fcfc3a6b7060390435dbed33d48e84ec77ee73a5df57300de8d5"} Sep 11 14:13:04.762847 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.762729 2519 scope.go:117] "RemoveContainer" containerID="1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27" Sep 11 14:13:04.764196 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.764178 2519 generic.go:356] "Generic (PLEG): container finished" podID="619dcec7-56c1-45a8-9bed-b89435d1d652" containerID="3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec" exitCode=0 Sep 11 14:13:04.764304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.764212 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" event={"ID":"619dcec7-56c1-45a8-9bed-b89435d1d652","Type":"ContainerDied","Data":"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec"} Sep 11 14:13:04.764304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.764233 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" Sep 11 14:13:04.764304 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.764247 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z" event={"ID":"619dcec7-56c1-45a8-9bed-b89435d1d652","Type":"ContainerDied","Data":"f22270e89979e5c80e9359826aca57f7cb55db6416c4b72e28574472f9d2d73b"} Sep 11 14:13:04.771722 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.771706 2519 scope.go:117] "RemoveContainer" containerID="1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27" Sep 11 14:13:04.772001 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:13:04.771969 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27\": container with ID starting with 1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27 not found: ID does not exist" containerID="1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27" Sep 11 14:13:04.772050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.772009 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27"} err="failed to get container status \"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27\": rpc error: code = NotFound desc = could not find container \"1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27\": container with ID starting with 1fd9807c7dbb92eaa259a108e22da792e0e777ada9912a82497d2af642c10c27 not found: ID does not exist" Sep 11 14:13:04.772050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.772026 2519 scope.go:117] "RemoveContainer" containerID="3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec" Sep 11 14:13:04.780110 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.780094 2519 scope.go:117] "RemoveContainer" containerID="3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec" Sep 11 14:13:04.780282 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.780259 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:13:04.780398 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:13:04.780382 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec\": container with ID starting with 3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec not found: ID does not exist" containerID="3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec" Sep 11 14:13:04.780435 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.780404 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec"} err="failed to get container status \"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec\": rpc error: code = NotFound desc = could not find container \"3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec\": container with ID starting with 3dd038f3f7f9b75694eea4b5ffa3e4fc2293ecee4e6114ea499fb6e18cf391ec not found: ID does not exist" Sep 11 14:13:04.785931 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.785910 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01faa9-5d9f5fb4cd-dkhgg"] Sep 11 14:13:04.795359 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.795339 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:13:04.800333 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:04.800314 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01faa9-55fd5674c-nc44z"] Sep 11 14:13:06.692104 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:06.692061 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619dcec7-56c1-45a8-9bed-b89435d1d652" path="/var/lib/kubelet/pods/619dcec7-56c1-45a8-9bed-b89435d1d652/volumes" Sep 11 14:13:06.692542 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:06.692524 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4561b9a-3861-443f-9fac-04c21591c3ed" path="/var/lib/kubelet/pods/f4561b9a-3861-443f-9fac-04c21591c3ed/volumes" Sep 11 14:13:10.573097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:10.573065 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:10.673012 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:10.672984 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:10.853234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:10.853201 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:21.328347 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:21.328295 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:22.289904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:22.289873 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:22.714965 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:22.714902 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:23.051846 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:23.051750 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:23.241664 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:23.241632 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:24.677263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:24.677204 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:26.106073 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:26.106032 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:26.958966 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:26.958929 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.293113 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.293005 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.397729 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.397697 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.495752 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.495688 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.831273 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.831222 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.927365 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.927332 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:27.954493 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.954463 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:13:27.954977 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.954958 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4561b9a-3861-443f-9fac-04c21591c3ed" containerName="payload-pre-processing" Sep 11 14:13:27.955071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.954979 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4561b9a-3861-443f-9fac-04c21591c3ed" containerName="payload-pre-processing" Sep 11 14:13:27.955071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955009 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1362cd8-f1b9-41ea-865b-3c9fd197c081" containerName="revoke-keys" Sep 11 14:13:27.955071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955020 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1362cd8-f1b9-41ea-865b-3c9fd197c081" containerName="revoke-keys" Sep 11 14:13:27.955071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955032 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619dcec7-56c1-45a8-9bed-b89435d1d652" containerName="payload-processing" Sep 11 14:13:27.955071 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955041 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="619dcec7-56c1-45a8-9bed-b89435d1d652" containerName="payload-processing" Sep 11 14:13:27.955330 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955145 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1362cd8-f1b9-41ea-865b-3c9fd197c081" containerName="revoke-keys" Sep 11 14:13:27.955330 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955161 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="619dcec7-56c1-45a8-9bed-b89435d1d652" containerName="payload-processing" Sep 11 14:13:27.955330 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.955175 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4561b9a-3861-443f-9fac-04c21591c3ed" containerName="payload-pre-processing" Sep 11 14:13:27.958441 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.958420 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:27.960819 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.960798 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e3622e\"" Sep 11 14:13:27.968781 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.968755 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:13:27.987098 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.987067 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58h4\" (UniqueName: \"kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:27.987230 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:27.987195 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.017609 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.017560 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:13:28.021301 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.021282 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.023642 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.023624 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e3622e-dockercfg-hksjh\"" Sep 11 14:13:28.031909 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.031883 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:13:28.088215 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.088181 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.088402 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.088237 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztkd\" (UniqueName: \"kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.088402 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.088265 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.088402 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.088327 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58h4\" (UniqueName: \"kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.089034 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.089012 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.096367 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.096333 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58h4\" (UniqueName: \"kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4\") pod \"payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.189083 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.189044 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.189298 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.189107 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztkd\" (UniqueName: \"kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.189827 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.189788 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.196953 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.196923 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztkd\" (UniqueName: \"kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd\") pod \"payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.269989 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.269953 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.332215 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.332176 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.544641 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.544616 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:13:28.546417 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:13:28.546394 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd677ab7f_1aa5_4817_b994_663a615990c0.slice/crio-b4e2294b5d67c7a36681ff79be2129dd08b41be8dbcf324e1529634a824daef2 WatchSource:0}: Error finding container b4e2294b5d67c7a36681ff79be2129dd08b41be8dbcf324e1529634a824daef2: Status 404 returned error can't find the container with id b4e2294b5d67c7a36681ff79be2129dd08b41be8dbcf324e1529634a824daef2 Sep 11 14:13:28.615232 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.615202 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:13:28.615943 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:13:28.615910 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab95b221_d9ec_44c0_99a9_afbe0fc88766.slice/crio-1faec406737dedefcdd1400cf2d9db74582c6bdbe7915fdba9db08591157fd1f WatchSource:0}: Error finding container 1faec406737dedefcdd1400cf2d9db74582c6bdbe7915fdba9db08591157fd1f: Status 404 returned error can't find the container with id 1faec406737dedefcdd1400cf2d9db74582c6bdbe7915fdba9db08591157fd1f Sep 11 14:13:28.897600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.897466 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" event={"ID":"ab95b221-d9ec-44c0-99a9-afbe0fc88766","Type":"ContainerStarted","Data":"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606"} Sep 11 14:13:28.897600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.897540 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" event={"ID":"ab95b221-d9ec-44c0-99a9-afbe0fc88766","Type":"ContainerStarted","Data":"1faec406737dedefcdd1400cf2d9db74582c6bdbe7915fdba9db08591157fd1f"} Sep 11 14:13:28.897600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.897589 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:28.899505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.899477 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" event={"ID":"d677ab7f-1aa5-4817-b994-663a615990c0","Type":"ContainerStarted","Data":"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3"} Sep 11 14:13:28.899662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.899528 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" event={"ID":"d677ab7f-1aa5-4817-b994-663a615990c0","Type":"ContainerStarted","Data":"b4e2294b5d67c7a36681ff79be2129dd08b41be8dbcf324e1529634a824daef2"} Sep 11 14:13:28.899662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.899559 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:28.914879 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.914831 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" podStartSLOduration=0.914816945 podStartE2EDuration="914.816945ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:28.912830426 +0000 UTC m=+1476.667145316" watchObservedRunningTime="2026-09-11 14:13:28.914816945 +0000 UTC m=+1476.669131835" Sep 11 14:13:28.935594 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:28.932038 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" podStartSLOduration=1.932016006 podStartE2EDuration="1.932016006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:28.927423467 +0000 UTC m=+1476.681738359" watchObservedRunningTime="2026-09-11 14:13:28.932016006 +0000 UTC m=+1476.686330899" Sep 11 14:13:35.483943 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:35.483889 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:39.907677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:39.907645 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:13:39.908058 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:39.907820 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:13:51.120561 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:51.120511 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:51.791810 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:51.791775 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:52.112141 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:52.112109 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:52.343129 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:52.343094 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:52.788900 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:52.788874 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:13:52.793873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:52.793848 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:13:55.359929 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:55.359882 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:55.457454 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:55.457411 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:56.619232 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:56.619200 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:56.683667 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:56.683633 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:56.795135 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:56.795094 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:57.124148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.124120 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:57.229505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.229475 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:57.254814 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.254781 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:13:57.258465 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.258442 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.260898 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.260873 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-68d0c7\"" Sep 11 14:13:57.261033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.261008 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-68d0c7-dockercfg-72526\"" Sep 11 14:13:57.267807 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.267785 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:13:57.273747 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.273723 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69sd\" (UniqueName: \"kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.273853 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.273769 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.375327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.375258 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69sd\" (UniqueName: \"kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.375327 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.375294 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.375992 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.375974 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.384908 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.384886 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69sd\" (UniqueName: \"kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd\") pod \"payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.571399 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.571363 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:57.845972 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:57.845942 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:13:57.847649 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:13:57.847621 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06411108_ff91_45fa_ba3e_8a6477f79527.slice/crio-2af9b66ce31fb618e07afbddfc8a53521052d9c490681f878748f3ef0798e789 WatchSource:0}: Error finding container 2af9b66ce31fb618e07afbddfc8a53521052d9c490681f878748f3ef0798e789: Status 404 returned error can't find the container with id 2af9b66ce31fb618e07afbddfc8a53521052d9c490681f878748f3ef0798e789 Sep 11 14:13:58.062245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:58.062207 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" event={"ID":"06411108-ff91-45fa-ba3e-8a6477f79527","Type":"ContainerStarted","Data":"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f"} Sep 11 14:13:58.062462 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:58.062272 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" event={"ID":"06411108-ff91-45fa-ba3e-8a6477f79527","Type":"ContainerStarted","Data":"2af9b66ce31fb618e07afbddfc8a53521052d9c490681f878748f3ef0798e789"} Sep 11 14:13:58.062462 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:58.062419 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:13:58.078867 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:13:58.078825 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" podStartSLOduration=1.078810124 podStartE2EDuration="1.078810124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:58.076650705 +0000 UTC m=+1505.830965596" watchObservedRunningTime="2026-09-11 14:13:58.078810124 +0000 UTC m=+1505.833125015" Sep 11 14:14:09.070024 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:09.069993 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:14:20.376918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:20.376884 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:25.493183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:25.493122 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log" Sep 11 14:14:25.624797 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:25.624752 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44_06411108-ff91-45fa-ba3e-8a6477f79527/payload-processing/0.log" Sep 11 14:14:36.492899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:36.492862 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:53.833769 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:14:53.833721 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:00.145216 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.145178 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv"] Sep 11 14:15:00.149342 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.149315 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr"] Sep 11 14:15:00.149512 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.149427 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" Sep 11 14:15:00.151710 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.151688 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:15:00.152660 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.152643 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6"] Sep 11 14:15:00.152774 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.152759 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" Sep 11 14:15:00.155972 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.155951 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" Sep 11 14:15:00.161848 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.161825 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" Sep 11 14:15:00.163521 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.163490 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv"] Sep 11 14:15:00.167795 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.167763 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr"] Sep 11 14:15:00.173673 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.173650 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6"] Sep 11 14:15:00.175421 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.175397 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" Sep 11 14:15:00.220713 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.220676 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" Sep 11 14:15:00.452036 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.452011 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv"] Sep 11 14:15:00.453623 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:15:00.453596 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ad016c_4067_481f_b998_e37b50f52e8d.slice/crio-8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e WatchSource:0}: Error finding container 8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e: Status 404 returned error can't find the container with id 8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e Sep 11 14:15:00.464264 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.464236 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr"] Sep 11 14:15:00.468960 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:15:00.468935 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aedf74_dcc1_4d00_9365_edc2bea9a68c.slice/crio-77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41 WatchSource:0}: Error finding container 77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41: Status 404 returned error can't find the container with id 77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41 Sep 11 14:15:00.511063 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:00.510943 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6"] Sep 11 14:15:00.513776 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:15:00.513746 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9409c715_c899_49eb_a0f7_616ef1cf20b0.slice/crio-917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a WatchSource:0}: Error finding container 917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a: Status 404 returned error can't find the container with id 917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a Sep 11 14:15:01.411145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.411107 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" event={"ID":"98aedf74-dcc1-4d00-9365-edc2bea9a68c","Type":"ContainerStarted","Data":"b471b3aa1bfa6310b1030b9b5ce04d9fdbdcc3315fc535874984a7626af428f1"} Sep 11 14:15:01.411570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.411177 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" event={"ID":"98aedf74-dcc1-4d00-9365-edc2bea9a68c","Type":"ContainerStarted","Data":"77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41"} Sep 11 14:15:01.412704 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.412676 2519 generic.go:356] "Generic (PLEG): container finished" podID="02ad016c-4067-481f-b998-e37b50f52e8d" containerID="f9eb6cb926640c51ac9c6011526a468a93e0f06a53502651ae8e5e8dbf482335" exitCode=0 Sep 11 14:15:01.412834 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.412756 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" event={"ID":"02ad016c-4067-481f-b998-e37b50f52e8d","Type":"ContainerDied","Data":"f9eb6cb926640c51ac9c6011526a468a93e0f06a53502651ae8e5e8dbf482335"} Sep 11 14:15:01.412834 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.412802 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" event={"ID":"02ad016c-4067-481f-b998-e37b50f52e8d","Type":"ContainerStarted","Data":"8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e"} Sep 11 14:15:01.414275 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.414254 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" event={"ID":"9409c715-c899-49eb-a0f7-616ef1cf20b0","Type":"ContainerStarted","Data":"5a70dcef080420b67599813dfb052008f53ab212329d78613ed853e2890b4f91"} Sep 11 14:15:01.414343 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.414295 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" event={"ID":"9409c715-c899-49eb-a0f7-616ef1cf20b0","Type":"ContainerStarted","Data":"917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a"} Sep 11 14:15:01.427142 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.427098 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" podStartSLOduration=1.4270837840000001 podStartE2EDuration="1.427083784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.425442579 +0000 UTC m=+1569.179757469" watchObservedRunningTime="2026-09-11 14:15:01.427083784 +0000 UTC m=+1569.181398674" Sep 11 14:15:01.440467 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:01.440414 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" podStartSLOduration=1.440398449 podStartE2EDuration="1.440398449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.439094023 +0000 UTC m=+1569.193408914" watchObservedRunningTime="2026-09-11 14:15:01.440398449 +0000 UTC m=+1569.194713355" Sep 11 14:15:02.421437 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:02.421405 2519 generic.go:356] "Generic (PLEG): container finished" podID="9409c715-c899-49eb-a0f7-616ef1cf20b0" containerID="5a70dcef080420b67599813dfb052008f53ab212329d78613ed853e2890b4f91" exitCode=0 Sep 11 14:15:02.421857 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:02.421454 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" event={"ID":"9409c715-c899-49eb-a0f7-616ef1cf20b0","Type":"ContainerDied","Data":"5a70dcef080420b67599813dfb052008f53ab212329d78613ed853e2890b4f91"} Sep 11 14:15:02.423045 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:02.423022 2519 generic.go:356] "Generic (PLEG): container finished" podID="98aedf74-dcc1-4d00-9365-edc2bea9a68c" containerID="b471b3aa1bfa6310b1030b9b5ce04d9fdbdcc3315fc535874984a7626af428f1" exitCode=0 Sep 11 14:15:02.423157 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:02.423100 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" event={"ID":"98aedf74-dcc1-4d00-9365-edc2bea9a68c","Type":"ContainerDied","Data":"b471b3aa1bfa6310b1030b9b5ce04d9fdbdcc3315fc535874984a7626af428f1"} Sep 11 14:15:02.548639 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:02.548611 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" Sep 11 14:15:03.429386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:03.429348 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" Sep 11 14:15:03.429386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:03.429367 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv" event={"ID":"02ad016c-4067-481f-b998-e37b50f52e8d","Type":"ContainerDied","Data":"8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e"} Sep 11 14:15:03.429896 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:03.429440 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0d7f435d64438b6632078cb4f90e7c606272e912b83f4448d29f435322040e" Sep 11 14:15:03.581688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:03.581660 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" Sep 11 14:15:03.584781 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:03.584757 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" Sep 11 14:15:04.436568 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.436542 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" Sep 11 14:15:04.437017 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.436538 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-q9gt6" event={"ID":"9409c715-c899-49eb-a0f7-616ef1cf20b0","Type":"ContainerDied","Data":"917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a"} Sep 11 14:15:04.437017 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.436678 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917d05e3bb61e6a427da58cbbed865846013df586a512567972f7f97773ea19a" Sep 11 14:15:04.438016 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.437987 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" event={"ID":"98aedf74-dcc1-4d00-9365-edc2bea9a68c","Type":"ContainerDied","Data":"77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41"} Sep 11 14:15:04.438121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.438021 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr" Sep 11 14:15:04.438121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:04.438029 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77842ddfe9001287d24384be4fffc7676b824f49d6a05b4e070e4b2f7db93d41" Sep 11 14:15:11.482015 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:11.481981 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh_d677ab7f-1aa5-4817-b994-663a615990c0/payload-processing/0.log" Sep 11 14:15:11.605200 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:11.605172 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log" Sep 11 14:15:11.713294 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:11.713266 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44_06411108-ff91-45fa-ba3e-8a6477f79527/payload-processing/0.log" Sep 11 14:15:11.985002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:11.984969 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:12.298812 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:12.298721 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:13.141050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:13.141018 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:15.654155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:15.654123 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:15.793326 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:15.793300 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:16.859292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:16.859254 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:16.958690 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:16.958642 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:17.088263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.088214 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:17.365593 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.365551 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:17.463852 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.463822 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:17.480778 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.480746 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:17.481297 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481279 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98aedf74-dcc1-4d00-9365-edc2bea9a68c" containerName="cleanup" Sep 11 14:15:17.481297 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481299 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="98aedf74-dcc1-4d00-9365-edc2bea9a68c" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481311 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9409c715-c899-49eb-a0f7-616ef1cf20b0" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481317 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="9409c715-c899-49eb-a0f7-616ef1cf20b0" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481340 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ad016c-4067-481f-b998-e37b50f52e8d" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481346 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ad016c-4067-481f-b998-e37b50f52e8d" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481416 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ad016c-4067-481f-b998-e37b50f52e8d" containerName="cleanup" Sep 11 14:15:17.481423 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481424 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="98aedf74-dcc1-4d00-9365-edc2bea9a68c" containerName="cleanup" Sep 11 14:15:17.481641 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.481432 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="9409c715-c899-49eb-a0f7-616ef1cf20b0" containerName="cleanup" Sep 11 14:15:17.484851 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.484829 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.488143 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.488120 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-24864d\"" Sep 11 14:15:17.488242 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.488125 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-24864d-dockercfg-kqnnf\"" Sep 11 14:15:17.496156 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.496131 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:17.563978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.563923 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.564145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.564046 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgtr\" (UniqueName: \"kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.665085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.664986 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.665085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.665048 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgtr\" (UniqueName: \"kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.665689 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.665668 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.673097 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.673058 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgtr\" (UniqueName: \"kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr\") pod \"payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:17.798348 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:17.798312 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:18.070750 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:18.070719 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:18.072543 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:15:18.072516 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac6def1_ff4f_49e3_a026_fcfdba7cc057.slice/crio-93ffc02e470cc2d97c370328a0f11de43a9115f1e2781f086c55dc1fe3052bc0 WatchSource:0}: Error finding container 93ffc02e470cc2d97c370328a0f11de43a9115f1e2781f086c55dc1fe3052bc0: Status 404 returned error can't find the container with id 93ffc02e470cc2d97c370328a0f11de43a9115f1e2781f086c55dc1fe3052bc0 Sep 11 14:15:18.516091 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:18.516058 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" event={"ID":"dac6def1-ff4f-49e3-a026-fcfdba7cc057","Type":"ContainerStarted","Data":"df7c7a88f33983c3650ec388e8bf0099c64267187e89946f38f4140da9274881"} Sep 11 14:15:18.516278 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:18.516105 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" event={"ID":"dac6def1-ff4f-49e3-a026-fcfdba7cc057","Type":"ContainerStarted","Data":"93ffc02e470cc2d97c370328a0f11de43a9115f1e2781f086c55dc1fe3052bc0"} Sep 11 14:15:18.516278 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:18.516160 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:18.532779 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:18.532728 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" podStartSLOduration=1.532713378 podStartE2EDuration="1.532713378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:18.531488957 +0000 UTC m=+1586.285803845" watchObservedRunningTime="2026-09-11 14:15:18.532713378 +0000 UTC m=+1586.287028269" Sep 11 14:15:21.890345 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:21.890311 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:21.952157 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:21.952117 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:29.526302 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:29.526272 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:40.745674 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:40.745642 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:41.393353 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.393315 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j"] Sep 11 14:15:41.397450 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.397430 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:41.400098 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.400067 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:15:41.402968 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.402903 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j"] Sep 11 14:15:41.500889 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.500850 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j\" (UID: \"ae474119-fec3-4d50-8a7a-71a8415f8369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:41.601746 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.601709 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j\" (UID: \"ae474119-fec3-4d50-8a7a-71a8415f8369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:41.602355 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.602322 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j\" (UID: \"ae474119-fec3-4d50-8a7a-71a8415f8369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:41.710041 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.709942 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:41.978830 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:41.978804 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j"] Sep 11 14:15:41.980592 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:15:41.980551 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae474119_fec3_4d50_8a7a_71a8415f8369.slice/crio-a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1 WatchSource:0}: Error finding container a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1: Status 404 returned error can't find the container with id a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1 Sep 11 14:15:42.661950 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:42.661908 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" event={"ID":"ae474119-fec3-4d50-8a7a-71a8415f8369","Type":"ContainerStarted","Data":"949f0ba5fa3248b1bdbc5c34139481c0d9f8a20061d82d44842c05ac6565f7b9"} Sep 11 14:15:42.662132 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:42.662020 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" event={"ID":"ae474119-fec3-4d50-8a7a-71a8415f8369","Type":"ContainerStarted","Data":"a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1"} Sep 11 14:15:42.678298 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:42.678235 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" podStartSLOduration=1.678218284 podStartE2EDuration="1.678218284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:42.67603655 +0000 UTC m=+1610.430351440" watchObservedRunningTime="2026-09-11 14:15:42.678218284 +0000 UTC m=+1610.432533205" Sep 11 14:15:43.668942 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:43.668910 2519 generic.go:356] "Generic (PLEG): container finished" podID="ae474119-fec3-4d50-8a7a-71a8415f8369" containerID="949f0ba5fa3248b1bdbc5c34139481c0d9f8a20061d82d44842c05ac6565f7b9" exitCode=0 Sep 11 14:15:43.669333 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:43.668957 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" event={"ID":"ae474119-fec3-4d50-8a7a-71a8415f8369","Type":"ContainerDied","Data":"949f0ba5fa3248b1bdbc5c34139481c0d9f8a20061d82d44842c05ac6565f7b9"} Sep 11 14:15:44.817457 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:44.817430 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:44.934911 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:44.934786 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca\") pod \"ae474119-fec3-4d50-8a7a-71a8415f8369\" (UID: \"ae474119-fec3-4d50-8a7a-71a8415f8369\") " Sep 11 14:15:44.935162 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:44.935137 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae474119-fec3-4d50-8a7a-71a8415f8369" (UID: "ae474119-fec3-4d50-8a7a-71a8415f8369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:45.036331 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:45.036290 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae474119-fec3-4d50-8a7a-71a8415f8369-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:15:45.681033 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:45.681001 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" Sep 11 14:15:45.681230 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:45.681004 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j" event={"ID":"ae474119-fec3-4d50-8a7a-71a8415f8369","Type":"ContainerDied","Data":"a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1"} Sep 11 14:15:45.681230 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:45.681138 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e47ddc2283a8cc1fc89ee360243c626e76d5b8973925f395bd7ed845d5d4e1" Sep 11 14:15:47.519458 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.519420 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:47.566798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.566765 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:47.567104 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.567071 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" podUID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" containerName="payload-pre-processing" containerID="cri-o://df7c7a88f33983c3650ec388e8bf0099c64267187e89946f38f4140da9274881" gracePeriod=30 Sep 11 14:15:47.619329 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.619292 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:47.694021 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.693992 2519 generic.go:356] "Generic (PLEG): container finished" podID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" containerID="df7c7a88f33983c3650ec388e8bf0099c64267187e89946f38f4140da9274881" exitCode=0 Sep 11 14:15:47.694154 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.694061 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" event={"ID":"dac6def1-ff4f-49e3-a026-fcfdba7cc057","Type":"ContainerDied","Data":"df7c7a88f33983c3650ec388e8bf0099c64267187e89946f38f4140da9274881"} Sep 11 14:15:47.719028 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.718997 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:47.746108 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.746074 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j"] Sep 11 14:15:47.748300 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.748275 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-24864d-mxj6j"] Sep 11 14:15:47.777586 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.777484 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:47.831571 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.831547 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:47.966115 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.966080 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxgtr\" (UniqueName: \"kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr\") pod \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " Sep 11 14:15:47.966115 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.966123 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume\") pod \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\" (UID: \"dac6def1-ff4f-49e3-a026-fcfdba7cc057\") " Sep 11 14:15:47.966534 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.966511 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dac6def1-ff4f-49e3-a026-fcfdba7cc057" (UID: "dac6def1-ff4f-49e3-a026-fcfdba7cc057"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:47.968387 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:47.968346 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr" (OuterVolumeSpecName: "kube-api-access-pxgtr") pod "dac6def1-ff4f-49e3-a026-fcfdba7cc057" (UID: "dac6def1-ff4f-49e3-a026-fcfdba7cc057"). InnerVolumeSpecName "kube-api-access-pxgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:15:48.067143 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.067052 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgtr\" (UniqueName: \"kubernetes.io/projected/dac6def1-ff4f-49e3-a026-fcfdba7cc057-kube-api-access-pxgtr\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:15:48.067143 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.067083 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac6def1-ff4f-49e3-a026-fcfdba7cc057-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:15:48.128269 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.128233 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:48.691930 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.691897 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae474119-fec3-4d50-8a7a-71a8415f8369" path="/var/lib/kubelet/pods/ae474119-fec3-4d50-8a7a-71a8415f8369/volumes" Sep 11 14:15:48.700692 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.700668 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" Sep 11 14:15:48.700692 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.700683 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497" event={"ID":"dac6def1-ff4f-49e3-a026-fcfdba7cc057","Type":"ContainerDied","Data":"93ffc02e470cc2d97c370328a0f11de43a9115f1e2781f086c55dc1fe3052bc0"} Sep 11 14:15:48.700838 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.700738 2519 scope.go:117] "RemoveContainer" containerID="df7c7a88f33983c3650ec388e8bf0099c64267187e89946f38f4140da9274881" Sep 11 14:15:48.716780 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.716754 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:48.718874 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:48.718851 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-24864d-58bb8654-28497"] Sep 11 14:15:50.692351 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:50.692315 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" path="/var/lib/kubelet/pods/dac6def1-ff4f-49e3-a026-fcfdba7cc057/volumes" Sep 11 14:15:53.973845 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:53.973805 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:54.770816 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:54.770781 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:54.850435 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:54.850403 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:58.043001 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:15:58.042962 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.461004 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.460967 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.489486 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.489451 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:16:13.489799 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.489766 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" podUID="d677ab7f-1aa5-4817-b994-663a615990c0" containerName="payload-processing" containerID="cri-o://600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3" gracePeriod=30 Sep 11 14:16:13.495961 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.495935 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:16:13.496223 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.496172 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" podUID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" containerName="payload-pre-processing" containerID="cri-o://514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606" gracePeriod=30 Sep 11 14:16:13.560591 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.560541 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.660875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.660842 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.761406 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.761376 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.770813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.770793 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:16:13.773850 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.773832 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:16:13.840100 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.840067 2519 generic.go:356] "Generic (PLEG): container finished" podID="d677ab7f-1aa5-4817-b994-663a615990c0" containerID="600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3" exitCode=0 Sep 11 14:16:13.840268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.840114 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" Sep 11 14:16:13.840268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.840143 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" event={"ID":"d677ab7f-1aa5-4817-b994-663a615990c0","Type":"ContainerDied","Data":"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3"} Sep 11 14:16:13.840268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.840192 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh" event={"ID":"d677ab7f-1aa5-4817-b994-663a615990c0","Type":"ContainerDied","Data":"b4e2294b5d67c7a36681ff79be2129dd08b41be8dbcf324e1529634a824daef2"} Sep 11 14:16:13.840268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.840218 2519 scope.go:117] "RemoveContainer" containerID="600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3" Sep 11 14:16:13.841677 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.841653 2519 generic.go:356] "Generic (PLEG): container finished" podID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" containerID="514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606" exitCode=0 Sep 11 14:16:13.841787 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.841685 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" event={"ID":"ab95b221-d9ec-44c0-99a9-afbe0fc88766","Type":"ContainerDied","Data":"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606"} Sep 11 14:16:13.841787 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.841703 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" Sep 11 14:16:13.841787 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.841734 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl" event={"ID":"ab95b221-d9ec-44c0-99a9-afbe0fc88766","Type":"ContainerDied","Data":"1faec406737dedefcdd1400cf2d9db74582c6bdbe7915fdba9db08591157fd1f"} Sep 11 14:16:13.849311 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.849295 2519 scope.go:117] "RemoveContainer" containerID="600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3" Sep 11 14:16:13.849598 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:16:13.849554 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3\": container with ID starting with 600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3 not found: ID does not exist" containerID="600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3" Sep 11 14:16:13.849598 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.849592 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3"} err="failed to get container status \"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3\": rpc error: code = NotFound desc = could not find container \"600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3\": container with ID starting with 600f441968cd3e1a8ca61c91ddbcd6557c8f09843c4de4a5e1800088ead822c3 not found: ID does not exist" Sep 11 14:16:13.849756 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.849609 2519 scope.go:117] "RemoveContainer" containerID="514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606" Sep 11 14:16:13.857654 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.857629 2519 scope.go:117] "RemoveContainer" containerID="514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606" Sep 11 14:16:13.857927 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:16:13.857907 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606\": container with ID starting with 514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606 not found: ID does not exist" containerID="514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606" Sep 11 14:16:13.857995 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.857933 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606"} err="failed to get container status \"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606\": rpc error: code = NotFound desc = could not find container \"514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606\": container with ID starting with 514edb48bc8c68249418c5e9dad6dbd08ccf61bb75aea8dfdfd1835417aa1606 not found: ID does not exist" Sep 11 14:16:13.906476 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906452 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lztkd\" (UniqueName: \"kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd\") pod \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " Sep 11 14:16:13.906661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906504 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume\") pod \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\" (UID: \"ab95b221-d9ec-44c0-99a9-afbe0fc88766\") " Sep 11 14:16:13.906661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906533 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume\") pod \"d677ab7f-1aa5-4817-b994-663a615990c0\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " Sep 11 14:16:13.906661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906651 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z58h4\" (UniqueName: \"kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4\") pod \"d677ab7f-1aa5-4817-b994-663a615990c0\" (UID: \"d677ab7f-1aa5-4817-b994-663a615990c0\") " Sep 11 14:16:13.906885 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906864 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d677ab7f-1aa5-4817-b994-663a615990c0" (UID: "d677ab7f-1aa5-4817-b994-663a615990c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:13.906947 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.906896 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab95b221-d9ec-44c0-99a9-afbe0fc88766" (UID: "ab95b221-d9ec-44c0-99a9-afbe0fc88766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:13.907028 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.907010 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d677ab7f-1aa5-4817-b994-663a615990c0-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:16:13.907079 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.907035 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab95b221-d9ec-44c0-99a9-afbe0fc88766-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:16:13.908684 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.908662 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4" (OuterVolumeSpecName: "kube-api-access-z58h4") pod "d677ab7f-1aa5-4817-b994-663a615990c0" (UID: "d677ab7f-1aa5-4817-b994-663a615990c0"). InnerVolumeSpecName "kube-api-access-z58h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:13.908747 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:13.908693 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd" (OuterVolumeSpecName: "kube-api-access-lztkd") pod "ab95b221-d9ec-44c0-99a9-afbe0fc88766" (UID: "ab95b221-d9ec-44c0-99a9-afbe0fc88766"). InnerVolumeSpecName "kube-api-access-lztkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:14.008521 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.008440 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58h4\" (UniqueName: \"kubernetes.io/projected/d677ab7f-1aa5-4817-b994-663a615990c0-kube-api-access-z58h4\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:16:14.008521 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.008472 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztkd\" (UniqueName: \"kubernetes.io/projected/ab95b221-d9ec-44c0-99a9-afbe0fc88766-kube-api-access-lztkd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:16:14.079164 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.079132 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:14.163179 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.163147 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:16:14.168996 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.168973 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3622e-579bfd5875-dqhqh"] Sep 11 14:16:14.177967 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.177944 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:16:14.181500 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.181474 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3622e-6fdd67f6f4-kxkbl"] Sep 11 14:16:14.691138 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.691103 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" path="/var/lib/kubelet/pods/ab95b221-d9ec-44c0-99a9-afbe0fc88766/volumes" Sep 11 14:16:14.691570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:14.691554 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d677ab7f-1aa5-4817-b994-663a615990c0" path="/var/lib/kubelet/pods/d677ab7f-1aa5-4817-b994-663a615990c0/volumes" Sep 11 14:16:20.100376 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:20.100341 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:24.858010 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:24.857977 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:53.414871 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:53.414832 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.434419 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:55.434377 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.981187 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:16:56.981151 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.551593 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:02.551554 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.651981 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:02.651948 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.818011 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:02.817919 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:03.177162 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177127 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng"] Sep 11 14:17:03.177739 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177718 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" containerName="payload-pre-processing" Sep 11 14:17:03.177739 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177740 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" containerName="payload-pre-processing" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177751 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d677ab7f-1aa5-4817-b994-663a615990c0" containerName="payload-processing" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177761 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="d677ab7f-1aa5-4817-b994-663a615990c0" containerName="payload-processing" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177786 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae474119-fec3-4d50-8a7a-71a8415f8369" containerName="revoke-keys" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177795 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae474119-fec3-4d50-8a7a-71a8415f8369" containerName="revoke-keys" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177823 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" containerName="payload-pre-processing" Sep 11 14:17:03.177873 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177831 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" containerName="payload-pre-processing" Sep 11 14:17:03.178140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177938 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae474119-fec3-4d50-8a7a-71a8415f8369" containerName="revoke-keys" Sep 11 14:17:03.178140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177956 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="d677ab7f-1aa5-4817-b994-663a615990c0" containerName="payload-processing" Sep 11 14:17:03.178140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177966 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac6def1-ff4f-49e3-a026-fcfdba7cc057" containerName="payload-pre-processing" Sep 11 14:17:03.178140 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.177975 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab95b221-d9ec-44c0-99a9-afbe0fc88766" containerName="payload-pre-processing" Sep 11 14:17:03.182100 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.182081 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:03.184666 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.184641 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:17:03.187093 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.187070 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng"] Sep 11 14:17:03.286625 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.286561 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng\" (UID: \"ad5c683d-ff40-4f94-873e-fdce2faf7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:03.387280 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.387238 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng\" (UID: \"ad5c683d-ff40-4f94-873e-fdce2faf7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:03.387918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.387898 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng\" (UID: \"ad5c683d-ff40-4f94-873e-fdce2faf7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:03.496230 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.496128 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:03.501269 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.501240 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:03.769675 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:03.769643 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng"] Sep 11 14:17:03.771033 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:17:03.771005 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5c683d_ff40_4f94_873e_fdce2faf7a6c.slice/crio-9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d WatchSource:0}: Error finding container 9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d: Status 404 returned error can't find the container with id 9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d Sep 11 14:17:04.123622 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:04.123568 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" event={"ID":"ad5c683d-ff40-4f94-873e-fdce2faf7a6c","Type":"ContainerStarted","Data":"28f998cba0cc62f1e6d3dfaa328a36c5cd97fd6d183809e1e6dcdb700c399e1b"} Sep 11 14:17:04.123792 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:04.123646 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" event={"ID":"ad5c683d-ff40-4f94-873e-fdce2faf7a6c","Type":"ContainerStarted","Data":"9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d"} Sep 11 14:17:04.138780 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:04.138730 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" podStartSLOduration=1.138712159 podStartE2EDuration="1.138712159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:04.138475216 +0000 UTC m=+1691.892790107" watchObservedRunningTime="2026-09-11 14:17:04.138712159 +0000 UTC m=+1691.893027067" Sep 11 14:17:05.130482 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:05.130446 2519 generic.go:356] "Generic (PLEG): container finished" podID="ad5c683d-ff40-4f94-873e-fdce2faf7a6c" containerID="28f998cba0cc62f1e6d3dfaa328a36c5cd97fd6d183809e1e6dcdb700c399e1b" exitCode=0 Sep 11 14:17:05.130921 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:05.130529 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" event={"ID":"ad5c683d-ff40-4f94-873e-fdce2faf7a6c","Type":"ContainerDied","Data":"28f998cba0cc62f1e6d3dfaa328a36c5cd97fd6d183809e1e6dcdb700c399e1b"} Sep 11 14:17:06.270038 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:06.270014 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:06.419196 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:06.419106 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca\") pod \"ad5c683d-ff40-4f94-873e-fdce2faf7a6c\" (UID: \"ad5c683d-ff40-4f94-873e-fdce2faf7a6c\") " Sep 11 14:17:06.419476 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:06.419451 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad5c683d-ff40-4f94-873e-fdce2faf7a6c" (UID: "ad5c683d-ff40-4f94-873e-fdce2faf7a6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:06.520478 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:06.520443 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5c683d-ff40-4f94-873e-fdce2faf7a6c-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:07.143105 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:07.143075 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" Sep 11 14:17:07.143105 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:07.143084 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng" event={"ID":"ad5c683d-ff40-4f94-873e-fdce2faf7a6c","Type":"ContainerDied","Data":"9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d"} Sep 11 14:17:07.143335 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:07.143131 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9659af9f3e2477a735eb78eba87189221e52795714465cdc831d4a2a89f6bd7d" Sep 11 14:17:13.875725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.875692 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr"] Sep 11 14:17:13.880638 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.880604 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-68d0c7-29818935-rk8hr"] Sep 11 14:17:13.887291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.887249 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:13.912225 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.912195 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:17:13.912420 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.912391 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" podUID="06411108-ff91-45fa-ba3e-8a6477f79527" containerName="payload-processing" containerID="cri-o://a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f" gracePeriod=30 Sep 11 14:17:13.982882 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:13.982849 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.084009 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.083972 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.102006 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.101965 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng"] Sep 11 14:17:14.105273 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.105241 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-68d0c7-k6jng"] Sep 11 14:17:14.150625 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.150514 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.174502 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.174475 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:17:14.181003 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.180978 2519 generic.go:356] "Generic (PLEG): container finished" podID="06411108-ff91-45fa-ba3e-8a6477f79527" containerID="a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f" exitCode=0 Sep 11 14:17:14.181138 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.181011 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" event={"ID":"06411108-ff91-45fa-ba3e-8a6477f79527","Type":"ContainerDied","Data":"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f"} Sep 11 14:17:14.181138 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.181031 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" Sep 11 14:17:14.181138 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.181092 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44" event={"ID":"06411108-ff91-45fa-ba3e-8a6477f79527","Type":"ContainerDied","Data":"2af9b66ce31fb618e07afbddfc8a53521052d9c490681f878748f3ef0798e789"} Sep 11 14:17:14.181138 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.181116 2519 scope.go:117] "RemoveContainer" containerID="a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f" Sep 11 14:17:14.183374 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.183352 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r69sd\" (UniqueName: \"kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd\") pod \"06411108-ff91-45fa-ba3e-8a6477f79527\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " Sep 11 14:17:14.183506 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.183406 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume\") pod \"06411108-ff91-45fa-ba3e-8a6477f79527\" (UID: \"06411108-ff91-45fa-ba3e-8a6477f79527\") " Sep 11 14:17:14.183831 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.183810 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06411108-ff91-45fa-ba3e-8a6477f79527" (UID: "06411108-ff91-45fa-ba3e-8a6477f79527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:14.185412 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.185391 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd" (OuterVolumeSpecName: "kube-api-access-r69sd") pod "06411108-ff91-45fa-ba3e-8a6477f79527" (UID: "06411108-ff91-45fa-ba3e-8a6477f79527"). InnerVolumeSpecName "kube-api-access-r69sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:14.192825 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.192805 2519 scope.go:117] "RemoveContainer" containerID="a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f" Sep 11 14:17:14.193273 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:17:14.193242 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f\": container with ID starting with a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f not found: ID does not exist" containerID="a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f" Sep 11 14:17:14.193431 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.193283 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f"} err="failed to get container status \"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f\": rpc error: code = NotFound desc = could not find container \"a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f\": container with ID starting with a9a7cde0cb9d5503ac84531759ca844acee83e25951508bf8d383b2752b8ee5f not found: ID does not exist" Sep 11 14:17:14.284352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.284320 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69sd\" (UniqueName: \"kubernetes.io/projected/06411108-ff91-45fa-ba3e-8a6477f79527-kube-api-access-r69sd\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:14.284352 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.284349 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06411108-ff91-45fa-ba3e-8a6477f79527-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:14.503584 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.503541 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:17:14.507693 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.507665 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.508506 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.508485 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-68d0c7-77fcccfb5f-5mh44"] Sep 11 14:17:14.693054 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.693018 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06411108-ff91-45fa-ba3e-8a6477f79527" path="/var/lib/kubelet/pods/06411108-ff91-45fa-ba3e-8a6477f79527/volumes" Sep 11 14:17:14.693306 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.693293 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98aedf74-dcc1-4d00-9365-edc2bea9a68c" path="/var/lib/kubelet/pods/98aedf74-dcc1-4d00-9365-edc2bea9a68c/volumes" Sep 11 14:17:14.693547 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:14.693536 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5c683d-ff40-4f94-873e-fdce2faf7a6c" path="/var/lib/kubelet/pods/ad5c683d-ff40-4f94-873e-fdce2faf7a6c/volumes" Sep 11 14:17:20.263016 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.262930 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:20.408064 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.408036 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:20.568461 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.568374 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:20.912891 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.912857 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q"] Sep 11 14:17:20.913554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913529 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06411108-ff91-45fa-ba3e-8a6477f79527" containerName="payload-processing" Sep 11 14:17:20.913688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913558 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="06411108-ff91-45fa-ba3e-8a6477f79527" containerName="payload-processing" Sep 11 14:17:20.913688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913594 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5c683d-ff40-4f94-873e-fdce2faf7a6c" containerName="revoke-keys" Sep 11 14:17:20.913688 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913604 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5c683d-ff40-4f94-873e-fdce2faf7a6c" containerName="revoke-keys" Sep 11 14:17:20.913844 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913730 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5c683d-ff40-4f94-873e-fdce2faf7a6c" containerName="revoke-keys" Sep 11 14:17:20.913844 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.913743 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="06411108-ff91-45fa-ba3e-8a6477f79527" containerName="payload-processing" Sep 11 14:17:20.919245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.919218 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:20.921371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.921342 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:17:20.922979 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.922955 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q"] Sep 11 14:17:20.951395 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:20.951360 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q\" (UID: \"b47f7ddd-391c-44be-84b6-b90a126b2896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:21.045376 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:21.045338 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:21.052713 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:21.052679 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q\" (UID: \"b47f7ddd-391c-44be-84b6-b90a126b2896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:21.053374 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:21.053352 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q\" (UID: \"b47f7ddd-391c-44be-84b6-b90a126b2896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:21.233405 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:21.233290 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:21.506150 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:21.506123 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q"] Sep 11 14:17:21.508470 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:17:21.508441 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f7ddd_391c_44be_84b6_b90a126b2896.slice/crio-2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b WatchSource:0}: Error finding container 2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b: Status 404 returned error can't find the container with id 2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b Sep 11 14:17:22.230259 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:22.230224 2519 generic.go:356] "Generic (PLEG): container finished" podID="b47f7ddd-391c-44be-84b6-b90a126b2896" containerID="346d4cc32b09427900990ba22540a31ebedcec1372b7f424779910c153892a67" exitCode=0 Sep 11 14:17:22.230440 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:22.230276 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" event={"ID":"b47f7ddd-391c-44be-84b6-b90a126b2896","Type":"ContainerDied","Data":"346d4cc32b09427900990ba22540a31ebedcec1372b7f424779910c153892a67"} Sep 11 14:17:22.230440 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:22.230332 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" event={"ID":"b47f7ddd-391c-44be-84b6-b90a126b2896","Type":"ContainerStarted","Data":"2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b"} Sep 11 14:17:23.369257 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:23.369230 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:23.474347 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:23.474307 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca\") pod \"b47f7ddd-391c-44be-84b6-b90a126b2896\" (UID: \"b47f7ddd-391c-44be-84b6-b90a126b2896\") " Sep 11 14:17:23.474672 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:23.474649 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b47f7ddd-391c-44be-84b6-b90a126b2896" (UID: "b47f7ddd-391c-44be-84b6-b90a126b2896"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:23.576094 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:23.576003 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47f7ddd-391c-44be-84b6-b90a126b2896-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:24.242120 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:24.242090 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" Sep 11 14:17:24.242433 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:24.242135 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q" event={"ID":"b47f7ddd-391c-44be-84b6-b90a126b2896","Type":"ContainerDied","Data":"2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b"} Sep 11 14:17:24.242433 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:24.242182 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba84c9fb35e6d4a8cba65f6ce532867f475f75aed7c0dbd7fe5904efc112c8b" Sep 11 14:17:31.461918 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.461877 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:31.494689 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.494654 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:17:31.494915 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.494890 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" podUID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" containerName="payload-processing" containerID="cri-o://fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4" gracePeriod=30 Sep 11 14:17:31.502505 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.502474 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:17:31.503875 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.502933 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" podUID="f2738fd9-4003-47a8-a29b-a65d94414aaf" containerName="payload-pre-processing" containerID="cri-o://6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105" gracePeriod=30 Sep 11 14:17:31.533990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.533960 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:31.593155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.593124 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:31.686261 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.686230 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q"] Sep 11 14:17:31.689523 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.689496 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:31.690661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.690638 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-932c97-lkl6q"] Sep 11 14:17:31.759044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.759011 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:31.780913 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.780891 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:17:31.783914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.783894 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:17:31.853559 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.853526 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx488\" (UniqueName: \"kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488\") pod \"f2738fd9-4003-47a8-a29b-a65d94414aaf\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " Sep 11 14:17:31.853776 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.853565 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume\") pod \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " Sep 11 14:17:31.853776 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.853658 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume\") pod \"f2738fd9-4003-47a8-a29b-a65d94414aaf\" (UID: \"f2738fd9-4003-47a8-a29b-a65d94414aaf\") " Sep 11 14:17:31.853776 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.853680 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjm5t\" (UniqueName: \"kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t\") pod \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\" (UID: \"86aac3c2-c69c-4b0a-b08a-0ee5e490416f\") " Sep 11 14:17:31.853988 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.853956 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86aac3c2-c69c-4b0a-b08a-0ee5e490416f" (UID: "86aac3c2-c69c-4b0a-b08a-0ee5e490416f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:31.854056 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.854037 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2738fd9-4003-47a8-a29b-a65d94414aaf" (UID: "f2738fd9-4003-47a8-a29b-a65d94414aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:31.855799 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.855778 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488" (OuterVolumeSpecName: "kube-api-access-gx488") pod "f2738fd9-4003-47a8-a29b-a65d94414aaf" (UID: "f2738fd9-4003-47a8-a29b-a65d94414aaf"). InnerVolumeSpecName "kube-api-access-gx488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:31.855876 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.855856 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t" (OuterVolumeSpecName: "kube-api-access-fjm5t") pod "86aac3c2-c69c-4b0a-b08a-0ee5e490416f" (UID: "86aac3c2-c69c-4b0a-b08a-0ee5e490416f"). InnerVolumeSpecName "kube-api-access-fjm5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:31.954790 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.954754 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx488\" (UniqueName: \"kubernetes.io/projected/f2738fd9-4003-47a8-a29b-a65d94414aaf-kube-api-access-gx488\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:31.954790 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.954783 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:31.954790 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.954795 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2738fd9-4003-47a8-a29b-a65d94414aaf-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:31.955052 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:31.954807 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjm5t\" (UniqueName: \"kubernetes.io/projected/86aac3c2-c69c-4b0a-b08a-0ee5e490416f-kube-api-access-fjm5t\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:32.095385 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.095350 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:32.288907 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.288870 2519 generic.go:356] "Generic (PLEG): container finished" podID="f2738fd9-4003-47a8-a29b-a65d94414aaf" containerID="6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105" exitCode=0 Sep 11 14:17:32.289107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.288916 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" Sep 11 14:17:32.289107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.288961 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" event={"ID":"f2738fd9-4003-47a8-a29b-a65d94414aaf","Type":"ContainerDied","Data":"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105"} Sep 11 14:17:32.289107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.289012 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t" event={"ID":"f2738fd9-4003-47a8-a29b-a65d94414aaf","Type":"ContainerDied","Data":"336a5b309b732b3b89935edf26f6113cd73db77aabe0576db63423b74f3836ba"} Sep 11 14:17:32.289107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.289035 2519 scope.go:117] "RemoveContainer" containerID="6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105" Sep 11 14:17:32.290401 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.290370 2519 generic.go:356] "Generic (PLEG): container finished" podID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" containerID="fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4" exitCode=0 Sep 11 14:17:32.290509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.290427 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" Sep 11 14:17:32.290509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.290447 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" event={"ID":"86aac3c2-c69c-4b0a-b08a-0ee5e490416f","Type":"ContainerDied","Data":"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4"} Sep 11 14:17:32.290509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.290499 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv" event={"ID":"86aac3c2-c69c-4b0a-b08a-0ee5e490416f","Type":"ContainerDied","Data":"d045d3cf8c3ef43b6df6e47523e90155a064a491373535ab3b3dac2eb23037af"} Sep 11 14:17:32.299956 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.299941 2519 scope.go:117] "RemoveContainer" containerID="6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105" Sep 11 14:17:32.300221 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:17:32.300189 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105\": container with ID starting with 6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105 not found: ID does not exist" containerID="6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105" Sep 11 14:17:32.300291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.300216 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105"} err="failed to get container status \"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105\": rpc error: code = NotFound desc = could not find container \"6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105\": container with ID starting with 6bd150f3022fa55be79d81d78e95c8dc55d51ff1b5c58f5e2fccb532ad9a7105 not found: ID does not exist" Sep 11 14:17:32.300291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.300234 2519 scope.go:117] "RemoveContainer" containerID="fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4" Sep 11 14:17:32.308380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.308360 2519 scope.go:117] "RemoveContainer" containerID="fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4" Sep 11 14:17:32.308665 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:17:32.308644 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4\": container with ID starting with fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4 not found: ID does not exist" containerID="fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4" Sep 11 14:17:32.308741 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.308670 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4"} err="failed to get container status \"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4\": rpc error: code = NotFound desc = could not find container \"fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4\": container with ID starting with fdccfe94543577e258e30aa54402e243c5e4a14f8bf65e9c5a40e3d431777ca4 not found: ID does not exist" Sep 11 14:17:32.312363 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.312341 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:17:32.316273 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.316252 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-932c97-865ccdd498-dgg5t"] Sep 11 14:17:32.325936 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.325913 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:17:32.330055 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.330035 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-932c97-854f8554d5-69bdv"] Sep 11 14:17:32.692419 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.692384 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" path="/var/lib/kubelet/pods/86aac3c2-c69c-4b0a-b08a-0ee5e490416f/volumes" Sep 11 14:17:32.692824 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.692762 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47f7ddd-391c-44be-84b6-b90a126b2896" path="/var/lib/kubelet/pods/b47f7ddd-391c-44be-84b6-b90a126b2896/volumes" Sep 11 14:17:32.693085 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:32.693072 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2738fd9-4003-47a8-a29b-a65d94414aaf" path="/var/lib/kubelet/pods/f2738fd9-4003-47a8-a29b-a65d94414aaf/volumes" Sep 11 14:17:38.289297 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.289259 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.587338 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.587303 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.652183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.652147 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.903974 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.903891 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8"] Sep 11 14:17:38.904317 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904305 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47f7ddd-391c-44be-84b6-b90a126b2896" containerName="revoke-keys" Sep 11 14:17:38.904361 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904319 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47f7ddd-391c-44be-84b6-b90a126b2896" containerName="revoke-keys" Sep 11 14:17:38.904361 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904334 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2738fd9-4003-47a8-a29b-a65d94414aaf" containerName="payload-pre-processing" Sep 11 14:17:38.904361 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904340 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2738fd9-4003-47a8-a29b-a65d94414aaf" containerName="payload-pre-processing" Sep 11 14:17:38.904361 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904352 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" containerName="payload-processing" Sep 11 14:17:38.904361 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904358 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" containerName="payload-processing" Sep 11 14:17:38.904516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904428 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aac3c2-c69c-4b0a-b08a-0ee5e490416f" containerName="payload-processing" Sep 11 14:17:38.904516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904435 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47f7ddd-391c-44be-84b6-b90a126b2896" containerName="revoke-keys" Sep 11 14:17:38.904516 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.904443 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2738fd9-4003-47a8-a29b-a65d94414aaf" containerName="payload-pre-processing" Sep 11 14:17:38.909180 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.909164 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:38.911678 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.911653 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zpmpt\"" Sep 11 14:17:38.914792 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:38.914765 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8"] Sep 11 14:17:39.023612 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.023538 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8\" (UID: \"cfd6159d-24d3-4042-a1a9-1085f214fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:39.124990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.124954 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8\" (UID: \"cfd6159d-24d3-4042-a1a9-1085f214fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:39.125597 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.125557 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8\" (UID: \"cfd6159d-24d3-4042-a1a9-1085f214fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:39.126849 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.126826 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:39.222506 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.222424 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:39.491742 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:39.491713 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8"] Sep 11 14:17:39.494375 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:17:39.494329 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd6159d_24d3_4042_a1a9_1085f214fcca.slice/crio-25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa WatchSource:0}: Error finding container 25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa: Status 404 returned error can't find the container with id 25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa Sep 11 14:17:40.345308 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:40.345270 2519 generic.go:356] "Generic (PLEG): container finished" podID="cfd6159d-24d3-4042-a1a9-1085f214fcca" containerID="480f52724df7a4974bdb3eaf15e1296fbeec604e74c7469492cf6d2d94aa7313" exitCode=0 Sep 11 14:17:40.345484 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:40.345359 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" event={"ID":"cfd6159d-24d3-4042-a1a9-1085f214fcca","Type":"ContainerDied","Data":"480f52724df7a4974bdb3eaf15e1296fbeec604e74c7469492cf6d2d94aa7313"} Sep 11 14:17:40.345484 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:40.345444 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" event={"ID":"cfd6159d-24d3-4042-a1a9-1085f214fcca","Type":"ContainerStarted","Data":"25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa"} Sep 11 14:17:41.482498 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:41.482468 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:41.548428 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:41.548397 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca\") pod \"cfd6159d-24d3-4042-a1a9-1085f214fcca\" (UID: \"cfd6159d-24d3-4042-a1a9-1085f214fcca\") " Sep 11 14:17:41.548776 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:41.548754 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfd6159d-24d3-4042-a1a9-1085f214fcca" (UID: "cfd6159d-24d3-4042-a1a9-1085f214fcca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:41.650228 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:41.650131 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd6159d-24d3-4042-a1a9-1085f214fcca-maas-api-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:42.358189 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:42.358160 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" Sep 11 14:17:42.358346 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:42.358204 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8" event={"ID":"cfd6159d-24d3-4042-a1a9-1085f214fcca","Type":"ContainerDied","Data":"25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa"} Sep 11 14:17:42.358346 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:42.358255 2519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d43300703f91f8611a2d83defd4223b1eb233157f99c0687d2d36a1f2a21fa" Sep 11 14:17:49.469689 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.469656 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv"] Sep 11 14:17:49.471044 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.471014 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.472775 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.472752 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-367517-29818935-79gzv"] Sep 11 14:17:49.502593 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.502547 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:17:49.502819 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.502789 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" podUID="45e6f873-510d-4c49-b729-2591508bed52" containerName="payload-processing" containerID="cri-o://c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461" gracePeriod=30 Sep 11 14:17:49.514998 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.514970 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:17:49.515181 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.515160 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" podUID="816f1ee5-762f-473f-928b-fd9baf9e4e47" containerName="payload-pre-processing" containerID="cri-o://b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911" gracePeriod=30 Sep 11 14:17:49.548380 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.548349 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.616358 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.616319 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.698872 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.698837 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8"] Sep 11 14:17:49.701990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.701946 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.702690 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.702670 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-367517-bhqm8"] Sep 11 14:17:49.770014 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.769979 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.787371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.787343 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:17:49.790404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.790378 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:17:49.932180 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932142 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume\") pod \"816f1ee5-762f-473f-928b-fd9baf9e4e47\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " Sep 11 14:17:49.932386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932207 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume\") pod \"45e6f873-510d-4c49-b729-2591508bed52\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " Sep 11 14:17:49.932386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932330 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqbzt\" (UniqueName: \"kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt\") pod \"45e6f873-510d-4c49-b729-2591508bed52\" (UID: \"45e6f873-510d-4c49-b729-2591508bed52\") " Sep 11 14:17:49.932386 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932354 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2r5w\" (UniqueName: \"kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w\") pod \"816f1ee5-762f-473f-928b-fd9baf9e4e47\" (UID: \"816f1ee5-762f-473f-928b-fd9baf9e4e47\") " Sep 11 14:17:49.932604 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932550 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "816f1ee5-762f-473f-928b-fd9baf9e4e47" (UID: "816f1ee5-762f-473f-928b-fd9baf9e4e47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:49.932656 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932569 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45e6f873-510d-4c49-b729-2591508bed52" (UID: "45e6f873-510d-4c49-b729-2591508bed52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:49.932774 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932752 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e6f873-510d-4c49-b729-2591508bed52-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:49.932815 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.932778 2519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816f1ee5-762f-473f-928b-fd9baf9e4e47-plugins-config-volume\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:49.934356 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.934333 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt" (OuterVolumeSpecName: "kube-api-access-mqbzt") pod "45e6f873-510d-4c49-b729-2591508bed52" (UID: "45e6f873-510d-4c49-b729-2591508bed52"). InnerVolumeSpecName "kube-api-access-mqbzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:49.934456 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:49.934380 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w" (OuterVolumeSpecName: "kube-api-access-g2r5w") pod "816f1ee5-762f-473f-928b-fd9baf9e4e47" (UID: "816f1ee5-762f-473f-928b-fd9baf9e4e47"). InnerVolumeSpecName "kube-api-access-g2r5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:50.034324 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.034233 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbzt\" (UniqueName: \"kubernetes.io/projected/45e6f873-510d-4c49-b729-2591508bed52-kube-api-access-mqbzt\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:50.034324 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.034265 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2r5w\" (UniqueName: \"kubernetes.io/projected/816f1ee5-762f-473f-928b-fd9baf9e4e47-kube-api-access-g2r5w\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:50.078134 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.078096 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:50.404046 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.404012 2519 generic.go:356] "Generic (PLEG): container finished" podID="45e6f873-510d-4c49-b729-2591508bed52" containerID="c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461" exitCode=0 Sep 11 14:17:50.404237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.404060 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" Sep 11 14:17:50.404237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.404085 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" event={"ID":"45e6f873-510d-4c49-b729-2591508bed52","Type":"ContainerDied","Data":"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461"} Sep 11 14:17:50.404237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.404136 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv" event={"ID":"45e6f873-510d-4c49-b729-2591508bed52","Type":"ContainerDied","Data":"800bc0324d0be86dae8d588bd1755e541f0e13a0c744ac6feb7a1536d6fe534e"} Sep 11 14:17:50.404237 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.404164 2519 scope.go:117] "RemoveContainer" containerID="c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461" Sep 11 14:17:50.405408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.405319 2519 generic.go:356] "Generic (PLEG): container finished" podID="816f1ee5-762f-473f-928b-fd9baf9e4e47" containerID="b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911" exitCode=0 Sep 11 14:17:50.405408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.405361 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" Sep 11 14:17:50.405408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.405397 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" event={"ID":"816f1ee5-762f-473f-928b-fd9baf9e4e47","Type":"ContainerDied","Data":"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911"} Sep 11 14:17:50.405549 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.405433 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb" event={"ID":"816f1ee5-762f-473f-928b-fd9baf9e4e47","Type":"ContainerDied","Data":"11e761a836c7c38ec081d888b35ea254b181034513b5cf9a9ac47c75a5a3722c"} Sep 11 14:17:50.417738 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.417716 2519 scope.go:117] "RemoveContainer" containerID="c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461" Sep 11 14:17:50.418099 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:17:50.418058 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461\": container with ID starting with c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461 not found: ID does not exist" containerID="c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461" Sep 11 14:17:50.418186 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.418107 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461"} err="failed to get container status \"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461\": rpc error: code = NotFound desc = could not find container \"c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461\": container with ID starting with c70ef07e26a8de6e8f5cf6d95040dd824c00535978137163b84bd8ad427cf461 not found: ID does not exist" Sep 11 14:17:50.418186 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.418131 2519 scope.go:117] "RemoveContainer" containerID="b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911" Sep 11 14:17:50.426539 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.426523 2519 scope.go:117] "RemoveContainer" containerID="b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911" Sep 11 14:17:50.426804 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:17:50.426785 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911\": container with ID starting with b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911 not found: ID does not exist" containerID="b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911" Sep 11 14:17:50.426867 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.426814 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911"} err="failed to get container status \"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911\": rpc error: code = NotFound desc = could not find container \"b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911\": container with ID starting with b7a976c553acc8a334c4c5cb5cc6e74dc31dbf336600eec2a62df93a83694911 not found: ID does not exist" Sep 11 14:17:50.432165 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.432141 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:17:50.440220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.440195 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-367517-78dc7d5d59-ltgcv"] Sep 11 14:17:50.451183 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.451159 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:17:50.454337 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.454317 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-367517-64f64cd6c4-59pzb"] Sep 11 14:17:50.690785 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.690696 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ad016c-4067-481f-b998-e37b50f52e8d" path="/var/lib/kubelet/pods/02ad016c-4067-481f-b998-e37b50f52e8d/volumes" Sep 11 14:17:50.691164 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.691073 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e6f873-510d-4c49-b729-2591508bed52" path="/var/lib/kubelet/pods/45e6f873-510d-4c49-b729-2591508bed52/volumes" Sep 11 14:17:50.691391 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.691378 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816f1ee5-762f-473f-928b-fd9baf9e4e47" path="/var/lib/kubelet/pods/816f1ee5-762f-473f-928b-fd9baf9e4e47/volumes" Sep 11 14:17:50.691740 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:50.691723 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd6159d-24d3-4042-a1a9-1085f214fcca" path="/var/lib/kubelet/pods/cfd6159d-24d3-4042-a1a9-1085f214fcca/volumes" Sep 11 14:17:55.741754 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:55.741720 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:56.145617 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:56.145558 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:56.208648 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:56.208609 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:56.380405 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:56.380354 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:56.478076 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:56.477986 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:59.349233 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.349202 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 14:17:59.349725 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.349402 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd946d98f-p59w5" podUID="97c2da0a-653c-4f0c-9b4d-41c193911017" containerName="manager" containerID="cri-o://f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61" gracePeriod=30 Sep 11 14:17:59.593920 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.593894 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 14:17:59.621440 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.621363 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs\") pod \"97c2da0a-653c-4f0c-9b4d-41c193911017\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " Sep 11 14:17:59.621440 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.621417 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmszp\" (UniqueName: \"kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp\") pod \"97c2da0a-653c-4f0c-9b4d-41c193911017\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " Sep 11 14:17:59.621682 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.621447 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") pod \"97c2da0a-653c-4f0c-9b4d-41c193911017\" (UID: \"97c2da0a-653c-4f0c-9b4d-41c193911017\") " Sep 11 14:17:59.623798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.623663 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97c2da0a-653c-4f0c-9b4d-41c193911017" (UID: "97c2da0a-653c-4f0c-9b4d-41c193911017"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:59.623798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.623753 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "97c2da0a-653c-4f0c-9b4d-41c193911017" (UID: "97c2da0a-653c-4f0c-9b4d-41c193911017"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:59.623798 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.623766 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp" (OuterVolumeSpecName: "kube-api-access-fmszp") pod "97c2da0a-653c-4f0c-9b4d-41c193911017" (UID: "97c2da0a-653c-4f0c-9b4d-41c193911017"). InnerVolumeSpecName "kube-api-access-fmszp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:59.723234 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.723205 2519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:59.723406 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.723243 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmszp\" (UniqueName: \"kubernetes.io/projected/97c2da0a-653c-4f0c-9b4d-41c193911017-kube-api-access-fmszp\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:17:59.723406 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:17:59.723271 2519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97c2da0a-653c-4f0c-9b4d-41c193911017-webhook-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:18:00.465565 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.465532 2519 generic.go:356] "Generic (PLEG): container finished" podID="97c2da0a-653c-4f0c-9b4d-41c193911017" containerID="f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61" exitCode=0 Sep 11 14:18:00.466082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.465601 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-p59w5" Sep 11 14:18:00.466082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.465626 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-p59w5" event={"ID":"97c2da0a-653c-4f0c-9b4d-41c193911017","Type":"ContainerDied","Data":"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61"} Sep 11 14:18:00.466082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.465696 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-p59w5" event={"ID":"97c2da0a-653c-4f0c-9b4d-41c193911017","Type":"ContainerDied","Data":"bcd99c280f9766f761bc8cffe405501e58accd7e65e2ca3f0f431bd6c485c89c"} Sep 11 14:18:00.466082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.465728 2519 scope.go:117] "RemoveContainer" containerID="f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61" Sep 11 14:18:00.475738 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.475722 2519 scope.go:117] "RemoveContainer" containerID="f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61" Sep 11 14:18:00.476003 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:18:00.475983 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61\": container with ID starting with f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61 not found: ID does not exist" containerID="f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61" Sep 11 14:18:00.476069 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.476014 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61"} err="failed to get container status \"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61\": rpc error: code = NotFound desc = could not find container \"f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61\": container with ID starting with f25906ca1a0f0ed8b9a05cb5179cabdfd55432e82388fa2a36d93ec555cfee61 not found: ID does not exist" Sep 11 14:18:00.487117 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.487094 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 14:18:00.488820 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.488799 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-p59w5"] Sep 11 14:18:00.691411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:00.691376 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c2da0a-653c-4f0c-9b4d-41c193911017" path="/var/lib/kubelet/pods/97c2da0a-653c-4f0c-9b4d-41c193911017/volumes" Sep 11 14:18:52.831827 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:52.831799 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:18:52.838377 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:52.838355 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:18:53.090902 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:18:53.090869 2519 scope.go:117] "RemoveContainer" containerID="6cf853d4443f1811bb0713a48885a84bee91adf74b3a3a8c51080a811defd7f7" Sep 11 14:19:01.305473 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305442 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305850 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c2da0a-653c-4f0c-9b4d-41c193911017" containerName="manager" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305863 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c2da0a-653c-4f0c-9b4d-41c193911017" containerName="manager" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305872 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816f1ee5-762f-473f-928b-fd9baf9e4e47" containerName="payload-pre-processing" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305877 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="816f1ee5-762f-473f-928b-fd9baf9e4e47" containerName="payload-pre-processing" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305884 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e6f873-510d-4c49-b729-2591508bed52" containerName="payload-processing" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305891 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e6f873-510d-4c49-b729-2591508bed52" containerName="payload-processing" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305902 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd6159d-24d3-4042-a1a9-1085f214fcca" containerName="revoke-keys" Sep 11 14:19:01.305978 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305907 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd6159d-24d3-4042-a1a9-1085f214fcca" containerName="revoke-keys" Sep 11 14:19:01.306272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305988 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="816f1ee5-762f-473f-928b-fd9baf9e4e47" containerName="payload-pre-processing" Sep 11 14:19:01.306272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.305997 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd6159d-24d3-4042-a1a9-1085f214fcca" containerName="revoke-keys" Sep 11 14:19:01.306272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.306004 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c2da0a-653c-4f0c-9b4d-41c193911017" containerName="manager" Sep 11 14:19:01.306272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.306010 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e6f873-510d-4c49-b729-2591508bed52" containerName="payload-processing" Sep 11 14:19:01.309227 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.309207 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.311684 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.311658 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 14:19:01.312543 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.312518 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 14:19:01.312543 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.312532 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hrdh6\"" Sep 11 14:19:01.312750 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.312542 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 14:19:01.318621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.318593 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:01.403371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.403268 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22th\" (UniqueName: \"kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.403371 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.403319 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.403644 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.403506 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.504508 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.504468 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.504753 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.504540 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22th\" (UniqueName: \"kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.504753 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.504610 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.507195 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.507166 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.507317 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.507166 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.512268 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.512238 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22th\" (UniqueName: \"kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th\") pod \"maas-controller-5cd946d98f-jhfwn\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.620382 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.620344 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:01.897291 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.897212 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:01.899608 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:19:01.899558 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e47448c_2916_458a_840d_b0ebf815ce0c.slice/crio-c89981a390487969bc163454e6f9f3e4d0d9a4fa69e5a52210d21536b561123c WatchSource:0}: Error finding container c89981a390487969bc163454e6f9f3e4d0d9a4fa69e5a52210d21536b561123c: Status 404 returned error can't find the container with id c89981a390487969bc163454e6f9f3e4d0d9a4fa69e5a52210d21536b561123c Sep 11 14:19:01.901058 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:01.901037 2519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:19:02.644622 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:02.644559 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:02.805364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:02.805265 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" event={"ID":"2e47448c-2916-458a-840d-b0ebf815ce0c","Type":"ContainerStarted","Data":"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6"} Sep 11 14:19:02.805364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:02.805315 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" event={"ID":"2e47448c-2916-458a-840d-b0ebf815ce0c","Type":"ContainerStarted","Data":"c89981a390487969bc163454e6f9f3e4d0d9a4fa69e5a52210d21536b561123c"} Sep 11 14:19:02.805364 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:02.805346 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:02.825660 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:02.825546 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" podStartSLOduration=1.223940535 podStartE2EDuration="1.82551946s" totalImagesPullingTime="601.578925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:02.822012402 +0000 UTC m=+1810.576327302" watchObservedRunningTime="2026-09-11 14:19:02.82551946 +0000 UTC m=+1810.579834352" Sep 11 14:19:12.934772 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:12.934716 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:12.990970 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:12.990940 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:13.812754 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:13.812725 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:23.052960 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.052914 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:23.154864 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.154831 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:23.155103 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.155058 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" podUID="2e47448c-2916-458a-840d-b0ebf815ce0c" containerName="manager" containerID="cri-o://b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6" gracePeriod=30 Sep 11 14:19:23.496841 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.496810 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:23.508990 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.508963 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert\") pod \"2e47448c-2916-458a-840d-b0ebf815ce0c\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " Sep 11 14:19:23.509145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.509028 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs\") pod \"2e47448c-2916-458a-840d-b0ebf815ce0c\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " Sep 11 14:19:23.509145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.509092 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22th\" (UniqueName: \"kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th\") pod \"2e47448c-2916-458a-840d-b0ebf815ce0c\" (UID: \"2e47448c-2916-458a-840d-b0ebf815ce0c\") " Sep 11 14:19:23.511060 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.511031 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2e47448c-2916-458a-840d-b0ebf815ce0c" (UID: "2e47448c-2916-458a-840d-b0ebf815ce0c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:19:23.511155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.511081 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th" (OuterVolumeSpecName: "kube-api-access-g22th") pod "2e47448c-2916-458a-840d-b0ebf815ce0c" (UID: "2e47448c-2916-458a-840d-b0ebf815ce0c"). InnerVolumeSpecName "kube-api-access-g22th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:23.511155 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.511086 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e47448c-2916-458a-840d-b0ebf815ce0c" (UID: "2e47448c-2916-458a-840d-b0ebf815ce0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:19:23.610349 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.610305 2519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.610349 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.610339 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22th\" (UniqueName: \"kubernetes.io/projected/2e47448c-2916-458a-840d-b0ebf815ce0c-kube-api-access-g22th\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.610349 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.610356 2519 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e47448c-2916-458a-840d-b0ebf815ce0c-webhook-cert\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.928721 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.928690 2519 generic.go:356] "Generic (PLEG): container finished" podID="2e47448c-2916-458a-840d-b0ebf815ce0c" containerID="b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6" exitCode=0 Sep 11 14:19:23.928904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.928742 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" Sep 11 14:19:23.928904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.928752 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" event={"ID":"2e47448c-2916-458a-840d-b0ebf815ce0c","Type":"ContainerDied","Data":"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6"} Sep 11 14:19:23.928904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.928794 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-jhfwn" event={"ID":"2e47448c-2916-458a-840d-b0ebf815ce0c","Type":"ContainerDied","Data":"c89981a390487969bc163454e6f9f3e4d0d9a4fa69e5a52210d21536b561123c"} Sep 11 14:19:23.928904 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.928816 2519 scope.go:117] "RemoveContainer" containerID="b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6" Sep 11 14:19:23.938272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.938256 2519 scope.go:117] "RemoveContainer" containerID="b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6" Sep 11 14:19:23.938522 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:19:23.938506 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6\": container with ID starting with b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6 not found: ID does not exist" containerID="b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6" Sep 11 14:19:23.938567 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.938530 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6"} err="failed to get container status \"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6\": rpc error: code = NotFound desc = could not find container \"b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6\": container with ID starting with b0fb2e0f15e8c0e10e489994e0ef496aaed9a6e32110261da8e06a923a35eee6 not found: ID does not exist" Sep 11 14:19:23.949081 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.949055 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:23.952969 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:23.952948 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-jhfwn"] Sep 11 14:19:24.651362 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.651326 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:24.691245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.691211 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e47448c-2916-458a-840d-b0ebf815ce0c" path="/var/lib/kubelet/pods/2e47448c-2916-458a-840d-b0ebf815ce0c/volumes" Sep 11 14:19:24.762656 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.762608 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd946d98f-6s9lf"] Sep 11 14:19:24.763092 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.763072 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e47448c-2916-458a-840d-b0ebf815ce0c" containerName="manager" Sep 11 14:19:24.763198 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.763094 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e47448c-2916-458a-840d-b0ebf815ce0c" containerName="manager" Sep 11 14:19:24.763251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.763222 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e47448c-2916-458a-840d-b0ebf815ce0c" containerName="manager" Sep 11 14:19:24.767783 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.767761 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.770192 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.770173 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 14:19:24.770757 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.770737 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hrdh6\"" Sep 11 14:19:24.770894 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.770761 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 14:19:24.771025 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.771009 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 14:19:24.775019 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.774993 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-6s9lf"] Sep 11 14:19:24.819987 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.819950 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5hq\" (UniqueName: \"kubernetes.io/projected/f58d9dce-ec9a-4333-8658-0661dad1fac8-kube-api-access-2x5hq\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.820170 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.819990 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-webhook-cert\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.820170 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.820112 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-metrics-certs\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.920630 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.920521 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-metrics-certs\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.920788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.920630 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5hq\" (UniqueName: \"kubernetes.io/projected/f58d9dce-ec9a-4333-8658-0661dad1fac8-kube-api-access-2x5hq\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.920788 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.920655 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-webhook-cert\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.923072 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.923040 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-metrics-certs\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.923177 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.923102 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f58d9dce-ec9a-4333-8658-0661dad1fac8-webhook-cert\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:24.929191 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:24.929171 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5hq\" (UniqueName: \"kubernetes.io/projected/f58d9dce-ec9a-4333-8658-0661dad1fac8-kube-api-access-2x5hq\") pod \"maas-controller-5cd946d98f-6s9lf\" (UID: \"f58d9dce-ec9a-4333-8658-0661dad1fac8\") " pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:25.080640 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.080593 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:25.350884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.350852 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd946d98f-6s9lf"] Sep 11 14:19:25.353064 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:19:25.353036 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58d9dce_ec9a_4333_8658_0661dad1fac8.slice/crio-578be05d7eb165c9f6d79c1a104238467ea3185add066cd86191cf631b2c4ee9 WatchSource:0}: Error finding container 578be05d7eb165c9f6d79c1a104238467ea3185add066cd86191cf631b2c4ee9: Status 404 returned error can't find the container with id 578be05d7eb165c9f6d79c1a104238467ea3185add066cd86191cf631b2c4ee9 Sep 11 14:19:25.941116 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.941080 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" event={"ID":"f58d9dce-ec9a-4333-8658-0661dad1fac8","Type":"ContainerStarted","Data":"84917f13bbe17a53c76c1c8148db0e55c8666d5f05954c8dc4dced7350e8e66a"} Sep 11 14:19:25.941489 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.941145 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:25.941489 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.941166 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" event={"ID":"f58d9dce-ec9a-4333-8658-0661dad1fac8","Type":"ContainerStarted","Data":"578be05d7eb165c9f6d79c1a104238467ea3185add066cd86191cf631b2c4ee9"} Sep 11 14:19:25.960550 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:25.960500 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" podStartSLOduration=1.6000090409999999 podStartE2EDuration="1.960484017s" totalImagesPullingTime="360.474976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:25.957925578 +0000 UTC m=+1833.712240480" watchObservedRunningTime="2026-09-11 14:19:25.960484017 +0000 UTC m=+1833.714798907" Sep 11 14:19:36.949120 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:36.949084 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd946d98f-6s9lf" Sep 11 14:19:53.116674 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:19:53.116640 2519 scope.go:117] "RemoveContainer" containerID="e8a8b6782af75b9865178869faf831973a0c81cd0ab37bc5f0598c9b7ddef93c" Sep 11 14:20:08.306185 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:20:08.306152 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:26.827360 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:21:26.827307 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:26.923186 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:21:26.923152 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:53.149912 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:21:53.149827 2519 scope.go:117] "RemoveContainer" containerID="949f0ba5fa3248b1bdbc5c34139481c0d9f8a20061d82d44842c05ac6565f7b9" Sep 11 14:21:53.158834 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:21:53.158810 2519 scope.go:117] "RemoveContainer" containerID="f9eb6cb926640c51ac9c6011526a468a93e0f06a53502651ae8e5e8dbf482335" Sep 11 14:21:53.166707 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:21:53.166526 2519 scope.go:117] "RemoveContainer" containerID="b471b3aa1bfa6310b1030b9b5ce04d9fdbdcc3315fc535874984a7626af428f1" Sep 11 14:22:19.194054 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.194006 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:22:19.197650 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.197632 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.201341 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.201316 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5327b004-kserve-self-signed-certs\"" Sep 11 14:22:19.216455 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.216434 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:22:19.290039 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290004 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.290199 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290053 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.290199 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290138 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.290280 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290222 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.290280 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290256 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.290347 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.290296 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cld2b\" (UniqueName: \"kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.302411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.302369 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:19.391704 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391662 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.391899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391749 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.391899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391790 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.391899 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391838 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cld2b\" (UniqueName: \"kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.392080 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391915 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.392080 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.391961 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.392193 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.392149 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.392324 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.392294 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.392450 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.392428 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.394488 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.394463 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.394488 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.394481 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.402181 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.402158 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cld2b\" (UniqueName: \"kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b\") pod \"e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.506750 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.506668 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:19.782101 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.782070 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:22:19.783675 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:22:19.783639 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6c198e3_0903_4999_a931_924b97ed3208.slice/crio-ce8f9e4e4d5bd50ef995f8d6e7e06cbc5bffee39a21db83469f229ff2fa6401b WatchSource:0}: Error finding container ce8f9e4e4d5bd50ef995f8d6e7e06cbc5bffee39a21db83469f229ff2fa6401b: Status 404 returned error can't find the container with id ce8f9e4e4d5bd50ef995f8d6e7e06cbc5bffee39a21db83469f229ff2fa6401b Sep 11 14:22:19.882061 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.882031 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" event={"ID":"e6c198e3-0903-4999-a931-924b97ed3208","Type":"ContainerStarted","Data":"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8"} Sep 11 14:22:19.882061 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.882070 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" event={"ID":"e6c198e3-0903-4999-a931-924b97ed3208","Type":"ContainerStarted","Data":"ce8f9e4e4d5bd50ef995f8d6e7e06cbc5bffee39a21db83469f229ff2fa6401b"} Sep 11 14:22:19.901280 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:19.901228 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" podStartSLOduration=0.901209418 podStartE2EDuration="901.209418ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:19.901170298 +0000 UTC m=+2007.655485201" watchObservedRunningTime="2026-09-11 14:22:19.901209418 +0000 UTC m=+2007.655524313" Sep 11 14:22:27.199924 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:27.199889 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:29.507482 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:29.507441 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:29.507884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:29.507508 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:29.523924 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:29.523893 2519 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:29.946345 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:29.946316 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:22:50.582002 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:50.581965 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:22:50.621406 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:50.621369 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:51.896079 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:22:51.896047 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:05.601314 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:05.601272 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" podUID="e6c198e3-0903-4999-a931-924b97ed3208" containerName="main" containerID="cri-o://5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8" gracePeriod=45 Sep 11 14:23:05.847353 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:05.847321 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:23:06.038076 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.037984 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038076 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038045 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038076 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038078 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038378 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038103 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038378 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038139 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cld2b\" (UniqueName: \"kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038378 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038188 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm\") pod \"e6c198e3-0903-4999-a931-924b97ed3208\" (UID: \"e6c198e3-0903-4999-a931-924b97ed3208\") " Sep 11 14:23:06.038378 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038274 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:06.038556 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038401 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home" (OuterVolumeSpecName: "home") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:06.038556 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038422 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:06.038709 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038625 2519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-tmp-dir\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.038709 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038639 2519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-model-cache\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.038709 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.038647 2519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-home\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.040485 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.040466 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:23:06.040549 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.040514 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm" (OuterVolumeSpecName: "dshm") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:06.040621 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.040604 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b" (OuterVolumeSpecName: "kube-api-access-cld2b") pod "e6c198e3-0903-4999-a931-924b97ed3208" (UID: "e6c198e3-0903-4999-a931-924b97ed3208"). InnerVolumeSpecName "kube-api-access-cld2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:06.134938 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.134899 2519 generic.go:356] "Generic (PLEG): container finished" podID="e6c198e3-0903-4999-a931-924b97ed3208" containerID="5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8" exitCode=0 Sep 11 14:23:06.135122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.134945 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" Sep 11 14:23:06.135122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.134984 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" event={"ID":"e6c198e3-0903-4999-a931-924b97ed3208","Type":"ContainerDied","Data":"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8"} Sep 11 14:23:06.135122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.135032 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn" event={"ID":"e6c198e3-0903-4999-a931-924b97ed3208","Type":"ContainerDied","Data":"ce8f9e4e4d5bd50ef995f8d6e7e06cbc5bffee39a21db83469f229ff2fa6401b"} Sep 11 14:23:06.135122 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.135051 2519 scope.go:117] "RemoveContainer" containerID="5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8" Sep 11 14:23:06.139957 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.139936 2519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c198e3-0903-4999-a931-924b97ed3208-tls-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.140082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.139960 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cld2b\" (UniqueName: \"kubernetes.io/projected/e6c198e3-0903-4999-a931-924b97ed3208-kube-api-access-cld2b\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.140082 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.139972 2519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c198e3-0903-4999-a931-924b97ed3208-dshm\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:06.144438 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.144422 2519 scope.go:117] "RemoveContainer" containerID="5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8" Sep 11 14:23:06.144838 ip-10-0-139-201 kubenswrapper[2519]: E0911 14:23:06.144816 2519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8\": container with ID starting with 5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8 not found: ID does not exist" containerID="5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8" Sep 11 14:23:06.144907 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.144847 2519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8"} err="failed to get container status \"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8\": rpc error: code = NotFound desc = could not find container \"5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8\": container with ID starting with 5e3028cc6a3f4166ba91ace1b1f965b6bea8d0fb9941809b0cb7ff8326e9ddd8 not found: ID does not exist" Sep 11 14:23:06.161914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.161882 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:23:06.173089 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.173063 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5327b004-kserve-85f45b8559-s29kn"] Sep 11 14:23:06.692709 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:06.692675 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c198e3-0903-4999-a931-924b97ed3208" path="/var/lib/kubelet/pods/e6c198e3-0903-4999-a931-924b97ed3208/volumes" Sep 11 14:23:13.144525 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.144489 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:13.144952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.144916 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c198e3-0903-4999-a931-924b97ed3208" containerName="main" Sep 11 14:23:13.144952 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.144928 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c198e3-0903-4999-a931-924b97ed3208" containerName="main" Sep 11 14:23:13.145029 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.144990 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c198e3-0903-4999-a931-924b97ed3208" containerName="main" Sep 11 14:23:13.149337 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.149318 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.152307 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.152285 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fjz8d\"" Sep 11 14:23:13.152307 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.152299 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 14:23:13.152548 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.152370 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:23:13.159167 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.159136 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:13.207971 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.207930 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mq8l\" (UniqueName: \"kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.208158 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.207996 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.208158 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.208036 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.208158 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.208113 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.309025 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.308989 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.309025 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.309029 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.309308 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.309090 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.309308 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.309119 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mq8l\" (UniqueName: \"kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.309779 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.309751 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.311864 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.311840 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.311965 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.311860 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.318171 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.318148 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mq8l\" (UniqueName: \"kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l\") pod \"maas-api-987c8478-slltd\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.463292 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.463189 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:13.745309 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:13.745154 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:13.748158 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:23:13.748118 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6d3674_74aa_4800_8222_5cfd05ee1185.slice/crio-eee029a9a478726d02f7ababae3a26e04d860f2ba1cdd97bdc7e45e6f11e45ec WatchSource:0}: Error finding container eee029a9a478726d02f7ababae3a26e04d860f2ba1cdd97bdc7e45e6f11e45ec: Status 404 returned error can't find the container with id eee029a9a478726d02f7ababae3a26e04d860f2ba1cdd97bdc7e45e6f11e45ec Sep 11 14:23:14.183570 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:14.183526 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" event={"ID":"8a6d3674-74aa-4800-8222-5cfd05ee1185","Type":"ContainerStarted","Data":"eee029a9a478726d02f7ababae3a26e04d860f2ba1cdd97bdc7e45e6f11e45ec"} Sep 11 14:23:16.195884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:16.195780 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" event={"ID":"8a6d3674-74aa-4800-8222-5cfd05ee1185","Type":"ContainerStarted","Data":"6f04991c8f3b8ed04a70979933288e72c430a6959d91bccd67ff8aeb4753bde2"} Sep 11 14:23:16.196314 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:16.195971 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:16.227714 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:16.227660 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" podStartSLOduration=1.065838558 podStartE2EDuration="3.227642387s" totalImagesPullingTime="2.161803829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:16.223468823 +0000 UTC m=+2063.977783714" watchObservedRunningTime="2026-09-11 14:23:16.227642387 +0000 UTC m=+2063.981957278" Sep 11 14:23:21.074927 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.074882 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:21.075450 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.075137 2519 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" podUID="8a6d3674-74aa-4800-8222-5cfd05ee1185" containerName="maas-api" containerID="cri-o://6f04991c8f3b8ed04a70979933288e72c430a6959d91bccd67ff8aeb4753bde2" gracePeriod=30 Sep 11 14:23:21.080263 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.080236 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:21.128755 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.128718 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds"] Sep 11 14:23:21.132903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.132878 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.147640 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.147605 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds"] Sep 11 14:23:21.189821 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.189783 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc93675d-eae3-4303-8b7b-9edd772e05b3-openshift-service-ca\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.189821 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.189827 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-metrics-certs\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.190010 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.189949 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-maas-api-tls\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.190010 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.189964 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898qv\" (UniqueName: \"kubernetes.io/projected/fc93675d-eae3-4303-8b7b-9edd772e05b3-kube-api-access-898qv\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.225368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.225334 2519 generic.go:356] "Generic (PLEG): container finished" podID="8a6d3674-74aa-4800-8222-5cfd05ee1185" containerID="6f04991c8f3b8ed04a70979933288e72c430a6959d91bccd67ff8aeb4753bde2" exitCode=0 Sep 11 14:23:21.225607 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.225403 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" event={"ID":"8a6d3674-74aa-4800-8222-5cfd05ee1185","Type":"ContainerDied","Data":"6f04991c8f3b8ed04a70979933288e72c430a6959d91bccd67ff8aeb4753bde2"} Sep 11 14:23:21.291396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.291356 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-maas-api-tls\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.291396 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.291397 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898qv\" (UniqueName: \"kubernetes.io/projected/fc93675d-eae3-4303-8b7b-9edd772e05b3-kube-api-access-898qv\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.291706 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.291518 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc93675d-eae3-4303-8b7b-9edd772e05b3-openshift-service-ca\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.291706 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.291559 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-metrics-certs\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.292161 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.292139 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc93675d-eae3-4303-8b7b-9edd772e05b3-openshift-service-ca\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.294018 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.293990 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-metrics-certs\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.294245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.294224 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc93675d-eae3-4303-8b7b-9edd772e05b3-maas-api-tls\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.302509 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.302484 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-898qv\" (UniqueName: \"kubernetes.io/projected/fc93675d-eae3-4303-8b7b-9edd772e05b3-kube-api-access-898qv\") pod \"maas-api-76677d8b8b-dcwds\" (UID: \"fc93675d-eae3-4303-8b7b-9edd772e05b3\") " pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.321713 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.321687 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:21.392554 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.392518 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls\") pod \"8a6d3674-74aa-4800-8222-5cfd05ee1185\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " Sep 11 14:23:21.392879 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.392602 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca\") pod \"8a6d3674-74aa-4800-8222-5cfd05ee1185\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " Sep 11 14:23:21.392879 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.392661 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs\") pod \"8a6d3674-74aa-4800-8222-5cfd05ee1185\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " Sep 11 14:23:21.392879 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.392718 2519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mq8l\" (UniqueName: \"kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l\") pod \"8a6d3674-74aa-4800-8222-5cfd05ee1185\" (UID: \"8a6d3674-74aa-4800-8222-5cfd05ee1185\") " Sep 11 14:23:21.393067 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.392943 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a6d3674-74aa-4800-8222-5cfd05ee1185" (UID: "8a6d3674-74aa-4800-8222-5cfd05ee1185"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:21.393107 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.393081 2519 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d3674-74aa-4800-8222-5cfd05ee1185-openshift-service-ca\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:21.395040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.395011 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a6d3674-74aa-4800-8222-5cfd05ee1185" (UID: "8a6d3674-74aa-4800-8222-5cfd05ee1185"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:23:21.395040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.395025 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a6d3674-74aa-4800-8222-5cfd05ee1185" (UID: "8a6d3674-74aa-4800-8222-5cfd05ee1185"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:23:21.395178 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.395014 2519 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l" (OuterVolumeSpecName: "kube-api-access-4mq8l") pod "8a6d3674-74aa-4800-8222-5cfd05ee1185" (UID: "8a6d3674-74aa-4800-8222-5cfd05ee1185"). InnerVolumeSpecName "kube-api-access-4mq8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:21.466272 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.466235 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:21.494442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.494403 2519 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-maas-api-tls\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:21.494442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.494438 2519 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6d3674-74aa-4800-8222-5cfd05ee1185-metrics-certs\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:21.494442 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.494447 2519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mq8l\" (UniqueName: \"kubernetes.io/projected/8a6d3674-74aa-4800-8222-5cfd05ee1185-kube-api-access-4mq8l\") on node \"ip-10-0-139-201.ec2.internal\" DevicePath \"\"" Sep 11 14:23:21.746903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:21.746872 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds"] Sep 11 14:23:21.748775 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:23:21.748748 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc93675d_eae3_4303_8b7b_9edd772e05b3.slice/crio-0b2d3a1fe816b0d95c0c2bc1ee16dc95656888956defa664c984ecb7d903b2a1 WatchSource:0}: Error finding container 0b2d3a1fe816b0d95c0c2bc1ee16dc95656888956defa664c984ecb7d903b2a1: Status 404 returned error can't find the container with id 0b2d3a1fe816b0d95c0c2bc1ee16dc95656888956defa664c984ecb7d903b2a1 Sep 11 14:23:22.233251 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.233205 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" event={"ID":"fc93675d-eae3-4303-8b7b-9edd772e05b3","Type":"ContainerStarted","Data":"94d7fda1e588b857d09cfc44b71317491e1f9b0340b87c569132c1e6671762af"} Sep 11 14:23:22.233726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.233272 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" event={"ID":"fc93675d-eae3-4303-8b7b-9edd772e05b3","Type":"ContainerStarted","Data":"0b2d3a1fe816b0d95c0c2bc1ee16dc95656888956defa664c984ecb7d903b2a1"} Sep 11 14:23:22.233726 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.233394 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:22.234559 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.234531 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" event={"ID":"8a6d3674-74aa-4800-8222-5cfd05ee1185","Type":"ContainerDied","Data":"eee029a9a478726d02f7ababae3a26e04d860f2ba1cdd97bdc7e45e6f11e45ec"} Sep 11 14:23:22.234661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.234608 2519 scope.go:117] "RemoveContainer" containerID="6f04991c8f3b8ed04a70979933288e72c430a6959d91bccd67ff8aeb4753bde2" Sep 11 14:23:22.234661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.234540 2519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-987c8478-slltd" Sep 11 14:23:22.254986 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.254940 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" podStartSLOduration=0.906714067 podStartE2EDuration="1.254923583s" totalImagesPullingTime="348.209516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:22.253024677 +0000 UTC m=+2070.007339568" watchObservedRunningTime="2026-09-11 14:23:22.254923583 +0000 UTC m=+2070.009238473" Sep 11 14:23:22.274245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.274211 2519 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:22.282460 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.282430 2519 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-987c8478-slltd"] Sep 11 14:23:22.692747 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:22.692704 2519 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6d3674-74aa-4800-8222-5cfd05ee1185" path="/var/lib/kubelet/pods/8a6d3674-74aa-4800-8222-5cfd05ee1185/volumes" Sep 11 14:23:28.246488 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:28.246458 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76677d8b8b-dcwds" Sep 11 14:23:43.547233 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:43.547197 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb644598-rb4x5_f91fa401-a2cd-4356-adb7-b2eb01cd962c/authorino/0.log" Sep 11 14:23:49.776662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:49.776629 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd946d98f-6s9lf_f58d9dce-ec9a-4333-8658-0661dad1fac8/manager/0.log" Sep 11 14:23:50.486813 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:50.486785 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85cf544d9c-gdtgr_cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca/copy-manifests/0.log" Sep 11 14:23:50.496507 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:50.496487 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85cf544d9c-gdtgr_cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca/manager/0.log" Sep 11 14:23:50.846113 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:50.846090 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76677d8b8b-dcwds_fc93675d-eae3-4303-8b7b-9edd772e05b3/maas-api/0.log" Sep 11 14:23:51.088411 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:51.088386 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818920-r5kxq_1cf9a634-ad29-4043-b115-ef68f521fa92/cleanup/0.log" Sep 11 14:23:51.211201 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:51.211127 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818935-q9gt6_9409c715-c899-49eb-a0f7-616ef1cf20b0/cleanup/0.log" Sep 11 14:23:51.331749 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:51.331726 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-mp4sj_83fcaf09-6ad9-496d-9bd4-43b240b168c7/postgres/0.log" Sep 11 14:23:52.144526 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:52.144493 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb644598-rb4x5_f91fa401-a2cd-4356-adb7-b2eb01cd962c/authorino/0.log" Sep 11 14:23:52.866295 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:52.866270 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:23:52.873929 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:52.873910 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bqvz8_36998c19-023d-4d86-8313-676fd0d6527b/ovn-acl-logging/0.log" Sep 11 14:23:53.200350 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:53.200271 2519 scope.go:117] "RemoveContainer" containerID="480f52724df7a4974bdb3eaf15e1296fbeec604e74c7469492cf6d2d94aa7313" Sep 11 14:23:53.208600 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:53.208569 2519 scope.go:117] "RemoveContainer" containerID="346d4cc32b09427900990ba22540a31ebedcec1372b7f424779910c153892a67" Sep 11 14:23:53.216086 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:53.216068 2519 scope.go:117] "RemoveContainer" containerID="28f998cba0cc62f1e6d3dfaa328a36c5cd97fd6d183809e1e6dcdb700c399e1b" Sep 11 14:23:53.554843 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:53.554765 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-846c6f95fb-254hr_38b08e90-2538-4641-bc75-26a575cc8cfe/kube-auth-proxy/0.log" Sep 11 14:23:53.972249 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:53.972215 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rbcpv_b42aca23-72e9-48f7-b863-2e10b4658e2f/payload-pre-processing/0.log" Sep 11 14:23:54.096884 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:54.096855 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log" Sep 11 14:23:54.717731 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:54.717699 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jh8ts_47f15cbc-b7a8-4b61-b21b-d6201974bcad/main/0.log" Sep 11 14:23:54.976809 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:23:54.976727 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zkx9c_905be396-da4c-4ae9-9e3c-f52ea0b5e809/main/0.log" Sep 11 14:24:00.898807 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:00.898764 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:01.000848 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:01.000809 2519 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:04.200547 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:04.200519 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rkqt7_21edc280-a387-48e2-a543-d24f2c5152cd/global-pull-secret-syncer/0.log" Sep 11 14:24:04.312216 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:04.312188 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8lngk_204076d3-c263-49ce-9747-eea120329029/konnectivity-agent/0.log" Sep 11 14:24:04.613145 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:04.613117 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-201.ec2.internal_b23ec63d9b3d220644ea926d481ee00f/haproxy/0.log" Sep 11 14:24:09.524358 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:09.524278 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb644598-rb4x5_f91fa401-a2cd-4356-adb7-b2eb01cd962c/authorino/0.log" Sep 11 14:24:11.560050 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.559951 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/alertmanager/0.log" Sep 11 14:24:11.621494 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.621465 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/config-reloader/0.log" Sep 11 14:24:11.653231 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.653202 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/kube-rbac-proxy-web/0.log" Sep 11 14:24:11.691733 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.691697 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/kube-rbac-proxy/0.log" Sep 11 14:24:11.760620 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.760591 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/kube-rbac-proxy-metric/0.log" Sep 11 14:24:11.831023 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.830957 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/prom-label-proxy/0.log" Sep 11 14:24:11.905115 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:11.905072 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e75d638c-16e1-41a2-92c8-cabb1f34c1fc/init-config-reloader/0.log" Sep 11 14:24:12.067807 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.067774 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rsnpb_48912ae9-262e-4f14-9e8f-ef1a13eb67bd/kube-state-metrics/0.log" Sep 11 14:24:12.141801 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.141773 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rsnpb_48912ae9-262e-4f14-9e8f-ef1a13eb67bd/kube-rbac-proxy-main/0.log" Sep 11 14:24:12.199774 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.199743 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rsnpb_48912ae9-262e-4f14-9e8f-ef1a13eb67bd/kube-rbac-proxy-self/0.log" Sep 11 14:24:12.257948 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.257922 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9c998d659-dgjw8_057201d7-c5b1-4380-98af-d92b2e4af41d/metrics-server/0.log" Sep 11 14:24:12.310859 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.310832 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-mns6w_f4d55d8c-ef95-4fac-ab1e-f6ef667e4a15/monitoring-plugin/0.log" Sep 11 14:24:12.381775 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.381747 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q427l_2ded9d1c-55fe-482c-8a34-bd8f57838d7a/node-exporter/0.log" Sep 11 14:24:12.424153 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.424076 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q427l_2ded9d1c-55fe-482c-8a34-bd8f57838d7a/kube-rbac-proxy/0.log" Sep 11 14:24:12.466967 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.466940 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q427l_2ded9d1c-55fe-482c-8a34-bd8f57838d7a/init-textfile/0.log" Sep 11 14:24:12.751088 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.751064 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw2zc_3250f149-7c06-43de-9376-b156cdc195ba/kube-rbac-proxy-main/0.log" Sep 11 14:24:12.775216 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.775181 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw2zc_3250f149-7c06-43de-9376-b156cdc195ba/kube-rbac-proxy-self/0.log" Sep 11 14:24:12.801955 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.801930 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tw2zc_3250f149-7c06-43de-9376-b156cdc195ba/openshift-state-metrics/0.log" Sep 11 14:24:12.853429 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.853385 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/prometheus/0.log" Sep 11 14:24:12.905698 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.905668 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/config-reloader/0.log" Sep 11 14:24:12.939965 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.939944 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/thanos-sidecar/0.log" Sep 11 14:24:12.983367 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:12.983336 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/kube-rbac-proxy-web/0.log" Sep 11 14:24:13.049040 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.048952 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/kube-rbac-proxy/0.log" Sep 11 14:24:13.093487 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.093451 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/kube-rbac-proxy-thanos/0.log" Sep 11 14:24:13.145220 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.145181 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b94830f2-bad7-47f8-9015-aa8db8b6d54d/init-config-reloader/0.log" Sep 11 14:24:13.192093 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.192067 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xwqzs_02e6a814-f1b8-40b3-b5e4-f83d3199a4e7/prometheus-operator/0.log" Sep 11 14:24:13.224422 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.224394 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xwqzs_02e6a814-f1b8-40b3-b5e4-f83d3199a4e7/kube-rbac-proxy/0.log" Sep 11 14:24:13.256121 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.256092 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-mn4vj_067a8f0b-2657-44fd-a978-152a55d372d8/prometheus-operator-admission-webhook/0.log" Sep 11 14:24:13.300408 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.300329 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-656d85656d-jkb62_fea163e9-a3cf-4bc5-b909-85c70b9cdb9a/telemeter-client/0.log" Sep 11 14:24:13.336771 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.336738 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-656d85656d-jkb62_fea163e9-a3cf-4bc5-b909-85c70b9cdb9a/reload/0.log" Sep 11 14:24:13.366951 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.366923 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-656d85656d-jkb62_fea163e9-a3cf-4bc5-b909-85c70b9cdb9a/kube-rbac-proxy/0.log" Sep 11 14:24:13.398948 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.398922 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/thanos-query/0.log" Sep 11 14:24:13.436777 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.436748 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/kube-rbac-proxy-web/0.log" Sep 11 14:24:13.486661 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.486631 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/kube-rbac-proxy/0.log" Sep 11 14:24:13.515083 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.515055 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/prom-label-proxy/0.log" Sep 11 14:24:13.552817 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.552744 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/kube-rbac-proxy-rules/0.log" Sep 11 14:24:13.579903 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.579878 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79fc48d8f6-9ptd5_e77445f2-60d8-436b-a2e7-89a26e2727cb/kube-rbac-proxy-metrics/0.log" Sep 11 14:24:13.958232 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.958199 2519 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7"] Sep 11 14:24:13.958662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.958629 2519 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6d3674-74aa-4800-8222-5cfd05ee1185" containerName="maas-api" Sep 11 14:24:13.958662 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.958641 2519 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6d3674-74aa-4800-8222-5cfd05ee1185" containerName="maas-api" Sep 11 14:24:13.958743 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.958723 2519 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6d3674-74aa-4800-8222-5cfd05ee1185" containerName="maas-api" Sep 11 14:24:13.962207 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.962179 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:13.964796 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.964780 2519 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdmhs\"/\"default-dockercfg-fw2z6\"" Sep 11 14:24:13.965359 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.965342 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdmhs\"/\"openshift-service-ca.crt\"" Sep 11 14:24:13.965412 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.965344 2519 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdmhs\"/\"kube-root-ca.crt\"" Sep 11 14:24:13.973705 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:13.973680 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7"] Sep 11 14:24:14.113950 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.113904 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-proc\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.114148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.114007 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc8c\" (UniqueName: \"kubernetes.io/projected/a2c66482-4471-448b-9539-e6bdd061d9e9-kube-api-access-5xc8c\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.114148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.114036 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-podres\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.114148 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.114055 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-sys\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.114281 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.114185 2519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-lib-modules\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215310 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-podres\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215349 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-sys\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215404 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215386 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-lib-modules\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215433 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-proc\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215471 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-sys\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215499 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-podres\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215512 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-lib-modules\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215484 2519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc8c\" (UniqueName: \"kubernetes.io/projected/a2c66482-4471-448b-9539-e6bdd061d9e9-kube-api-access-5xc8c\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.215715 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.215555 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2c66482-4471-448b-9539-e6bdd061d9e9-proc\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.226181 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.226161 2519 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc8c\" (UniqueName: \"kubernetes.io/projected/a2c66482-4471-448b-9539-e6bdd061d9e9-kube-api-access-5xc8c\") pod \"perf-node-gather-daemonset-zhtw7\" (UID: \"a2c66482-4471-448b-9539-e6bdd061d9e9\") " pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.273334 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.273296 2519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:14.548368 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:14.548340 2519 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7"] Sep 11 14:24:14.549511 ip-10-0-139-201 kubenswrapper[2519]: W0911 14:24:14.549478 2519 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2c66482_4471_448b_9539_e6bdd061d9e9.slice/crio-56a9d0207aa5028b8a4627540f0a078801e26e0e9b050fa2f04d8e4fff4de86d WatchSource:0}: Error finding container 56a9d0207aa5028b8a4627540f0a078801e26e0e9b050fa2f04d8e4fff4de86d: Status 404 returned error can't find the container with id 56a9d0207aa5028b8a4627540f0a078801e26e0e9b050fa2f04d8e4fff4de86d Sep 11 14:24:15.511316 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:15.511276 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" event={"ID":"a2c66482-4471-448b-9539-e6bdd061d9e9","Type":"ContainerStarted","Data":"9db55678d90ca273d7a2f92702c83e2121177016e32d36e6502156149ea55b93"} Sep 11 14:24:15.511733 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:15.511337 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:15.511733 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:15.511354 2519 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" event={"ID":"a2c66482-4471-448b-9539-e6bdd061d9e9","Type":"ContainerStarted","Data":"56a9d0207aa5028b8a4627540f0a078801e26e0e9b050fa2f04d8e4fff4de86d"} Sep 11 14:24:15.531804 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:15.531756 2519 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" podStartSLOduration=2.531740769 podStartE2EDuration="2.531740769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:15.53127849 +0000 UTC m=+2123.285593394" watchObservedRunningTime="2026-09-11 14:24:15.531740769 +0000 UTC m=+2123.286055659" Sep 11 14:24:15.997200 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:15.997172 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d76db8fd-xtdpn_1e382750-176b-4fe3-ad57-b919d4281589/console/0.log" Sep 11 14:24:17.983713 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:17.983676 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w25ml_9e83873e-486e-49b2-9db7-f879d795ae28/dns/0.log" Sep 11 14:24:18.014644 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:18.014615 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w25ml_9e83873e-486e-49b2-9db7-f879d795ae28/kube-rbac-proxy/0.log" Sep 11 14:24:18.114666 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:18.114632 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pdvmj_3b4a9107-64d8-45be-bfe9-ed28093f3c83/dns-node-resolver/0.log" Sep 11 14:24:18.813652 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:18.813619 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvrq9_b84a9e05-9178-4368-8744-1b49ee4ba041/node-ca/0.log" Sep 11 14:24:20.166613 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:20.166569 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-846c6f95fb-254hr_38b08e90-2538-4641-bc75-26a575cc8cfe/kube-auth-proxy/0.log" Sep 11 14:24:20.279566 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:20.279529 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rbcpv_b42aca23-72e9-48f7-b863-2e10b4658e2f/payload-pre-processing/0.log" Sep 11 14:24:20.321286 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:20.321253 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log" Sep 11 14:24:21.363398 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:21.363368 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzvrs_5b99e603-b615-4339-bded-ee7639b7f5ce/serve-healthcheck-canary/0.log" Sep 11 14:24:21.526379 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:21.526353 2519 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdmhs/perf-node-gather-daemonset-zhtw7" Sep 11 14:24:22.040347 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:22.040310 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jkd_3e744bc4-f55d-466a-9d34-95d2cc16bf2f/kube-rbac-proxy/0.log" Sep 11 14:24:22.073383 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:22.073351 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jkd_3e744bc4-f55d-466a-9d34-95d2cc16bf2f/exporter/0.log" Sep 11 14:24:22.100168 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:22.100140 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r8jkd_3e744bc4-f55d-466a-9d34-95d2cc16bf2f/extractor/0.log" Sep 11 14:24:25.051914 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.051881 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd946d98f-6s9lf_f58d9dce-ec9a-4333-8658-0661dad1fac8/manager/0.log" Sep 11 14:24:25.456405 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.456378 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-846c6f95fb-254hr_38b08e90-2538-4641-bc75-26a575cc8cfe/kube-auth-proxy/0.log" Sep 11 14:24:25.556358 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.556331 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rbcpv_b42aca23-72e9-48f7-b863-2e10b4658e2f/payload-pre-processing/0.log" Sep 11 14:24:25.581583 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.581546 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85cf544d9c-gdtgr_cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca/manager/0.log" Sep 11 14:24:25.597610 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.597589 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log" Sep 11 14:24:25.651206 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:25.651183 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85cf544d9c-gdtgr_cc07ec83-33bb-4aa9-ac49-98c3c3b7d6ca/copy-manifests/0.log" Sep 11 14:24:27.145405 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:27.145378 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8546dd6968-ng94s_48124b0a-fc95-4979-8a6c-50bff066a934/manager/0.log" Sep 11 14:24:27.222245 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:27.222214 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zdt7w_a78b44e4-e833-4f97-b0f1-ebdfd83e8979/openshift-lws-operator/0.log" Sep 11 14:24:27.541290 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:27.541206 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-846c6f95fb-254hr_38b08e90-2538-4641-bc75-26a575cc8cfe/kube-auth-proxy/0.log" Sep 11 14:24:27.649269 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:27.649234 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rbcpv_b42aca23-72e9-48f7-b863-2e10b4658e2f/payload-pre-processing/0.log" Sep 11 14:24:27.693612 ip-10-0-139-201 kubenswrapper[2519]: I0911 14:24:27.693563 2519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-z2s2j_a526c097-255f-4589-b80e-cd62afaa9128/payload-processing/0.log"