Sep 11 14:13:11.686886 ip-10-0-141-198 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 14:13:11.686895 ip-10-0-141-198 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 14:13:11.686902 ip-10-0-141-198 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 14:13:11.687066 ip-10-0-141-198 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 14:13:21.719407 ip-10-0-141-198 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 14:13:21.719421 ip-10-0-141-198 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2bcb1c6f3a1f47e3aa1f1d5936bb9823 -- Sep 11 14:14:57.315286 ip-10-0-141-198 systemd[1]: Starting Kubernetes Kubelet... Sep 11 14:14:57.754760 ip-10-0-141-198 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:57.754760 ip-10-0-141-198 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 14:14:57.754760 ip-10-0-141-198 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:57.754760 ip-10-0-141-198 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 14:14:57.754760 ip-10-0-141-198 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:57.755727 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.755636 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 14:14:57.759873 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759856 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.759873 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759874 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759877 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759883 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759888 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759892 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759895 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759899 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759902 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759905 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759908 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759910 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759914 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759917 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759927 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759929 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759944 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759947 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759949 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759952 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759955 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.759965 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759958 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759960 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759963 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759966 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759969 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759972 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759975 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759977 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759980 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759983 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759985 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759988 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759991 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759993 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759996 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.759998 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760000 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760003 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760005 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760008 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.760449 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760011 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760014 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760018 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760021 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760024 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760027 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760030 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760032 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760035 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760038 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760040 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760042 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760045 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760048 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760051 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760054 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760057 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760059 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760062 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.760952 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760065 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760068 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760070 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760073 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760075 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760077 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760080 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760083 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760085 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760088 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760090 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760093 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760096 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760098 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760101 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760104 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760108 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760111 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760114 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.761419 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760117 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760120 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760122 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760125 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760127 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760130 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760133 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760136 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760139 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760142 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760144 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760148 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760150 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760153 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760156 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760158 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760161 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760163 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760166 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760168 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.761898 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760918 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760925 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760928 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760945 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760948 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760957 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760960 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760962 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760966 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760968 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760971 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760973 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760976 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760978 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760981 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760984 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760987 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760990 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760994 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760997 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.760999 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.762750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761002 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761005 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761007 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761010 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761013 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761015 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761018 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761021 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761024 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761026 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761029 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761031 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761033 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761036 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761039 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761041 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761044 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761052 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761055 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761057 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.763678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761060 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761062 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761065 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761067 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761070 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761072 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761075 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761080 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761084 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761089 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761092 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761096 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761099 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761102 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761105 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761108 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761111 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761114 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761116 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.764200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761119 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761121 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761124 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761126 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761129 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761131 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761134 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761136 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761139 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761141 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761144 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761147 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761150 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761153 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761156 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761159 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761161 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761164 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761166 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.764783 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761169 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761172 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761175 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761178 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761180 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761184 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761187 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761189 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761192 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761194 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761197 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761200 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761202 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761205 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761208 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761210 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761213 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761216 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761219 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761222 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.765277 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761617 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761623 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761627 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761629 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761633 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761635 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761638 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761641 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761644 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761647 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761650 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761652 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761655 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761658 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761661 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761664 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761667 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761669 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761672 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761676 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.765809 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761679 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761682 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761684 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761687 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761690 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761693 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761695 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761698 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761701 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761704 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761706 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761709 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761712 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761714 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761717 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761720 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761722 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761726 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761728 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761731 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.766426 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761733 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761736 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761738 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761741 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761743 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761750 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761753 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761755 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761758 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761760 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761763 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761766 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761769 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761772 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761775 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761777 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761780 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761783 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761785 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761788 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.767049 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761790 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761793 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761796 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761798 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761801 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761804 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761807 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761810 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761815 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761818 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761821 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761824 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761827 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761829 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761832 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761835 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761837 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761841 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761844 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.767533 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761847 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761849 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761853 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761857 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761859 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761862 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761864 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761868 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761872 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761874 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761877 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761880 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761883 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761885 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761888 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761891 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761894 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761897 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761899 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.768160 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.761903 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762038 2571 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762116 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762129 2571 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762137 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762145 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762151 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762158 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762165 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762170 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762175 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762181 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762186 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762191 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762196 2571 flags.go:64] FLAG: --cgroup-root="" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762201 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762206 2571 flags.go:64] FLAG: --client-ca-file="" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762211 2571 flags.go:64] FLAG: --cloud-provider="external" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762216 2571 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762222 2571 flags.go:64] FLAG: --cluster-domain="" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762227 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762233 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762239 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762244 2571 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 14:14:57.768642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762250 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762254 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762260 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762266 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762271 2571 flags.go:64] FLAG: --contention-profiling="false" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762276 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762281 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762286 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762291 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762299 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762306 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762311 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762316 2571 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762322 2571 flags.go:64] FLAG: --enable-server="true" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762327 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762335 2571 flags.go:64] FLAG: --event-burst="100" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762340 2571 flags.go:64] FLAG: --event-qps="50" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762345 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762350 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762356 2571 flags.go:64] FLAG: --eviction-hard="" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762362 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762367 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762372 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762377 2571 flags.go:64] FLAG: --eviction-soft="" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762381 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 14:14:57.769266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762386 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762391 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762396 2571 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762401 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762406 2571 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762410 2571 flags.go:64] FLAG: --feature-gates="" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762416 2571 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762421 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762427 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762432 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762437 2571 flags.go:64] FLAG: --healthz-port="10248" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762442 2571 flags.go:64] FLAG: --help="false" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762446 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-141-198.ec2.internal" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762451 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762456 2571 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762461 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762467 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762475 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762481 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762486 2571 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762491 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762495 2571 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762500 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762505 2571 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 14:14:57.770005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762510 2571 flags.go:64] FLAG: --kube-reserved="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762515 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762519 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762524 2571 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762529 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762534 2571 flags.go:64] FLAG: --lock-file="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762539 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762543 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762548 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762557 2571 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762562 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762568 2571 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762572 2571 flags.go:64] FLAG: --logging-format="text" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762577 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762583 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762588 2571 flags.go:64] FLAG: --manifest-url="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762592 2571 flags.go:64] FLAG: --manifest-url-header="" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762600 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762605 2571 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762612 2571 flags.go:64] FLAG: --max-pods="110" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762617 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762622 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762626 2571 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762632 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762642 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 14:14:57.770597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762647 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762652 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762665 2571 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762670 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762676 2571 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762681 2571 flags.go:64] FLAG: --pod-cidr="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762686 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762695 2571 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762700 2571 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762705 2571 flags.go:64] FLAG: --pods-per-core="0" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762709 2571 flags.go:64] FLAG: --port="10250" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762715 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762720 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0911fef4f0f2a245e" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762725 2571 flags.go:64] FLAG: --qos-reserved="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762730 2571 flags.go:64] FLAG: --read-only-port="10255" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762735 2571 flags.go:64] FLAG: --register-node="true" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762739 2571 flags.go:64] FLAG: --register-with-taints="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762745 2571 flags.go:64] FLAG: --registry-burst="10" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762749 2571 flags.go:64] FLAG: --registry-qps="5" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762754 2571 flags.go:64] FLAG: --reserved-cpus="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762758 2571 flags.go:64] FLAG: --reserved-memory="" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762763 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762768 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762773 2571 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762777 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 14:14:57.771309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762782 2571 flags.go:64] FLAG: --runonce="false" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762787 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762792 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762798 2571 flags.go:64] FLAG: --seccomp-default="false" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762802 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762807 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762814 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762819 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762824 2571 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762829 2571 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762834 2571 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762839 2571 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762844 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762850 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762855 2571 flags.go:64] FLAG: --system-cgroups="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762859 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762870 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762875 2571 flags.go:64] FLAG: --tls-cert-file="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762880 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762885 2571 flags.go:64] FLAG: --tls-min-version="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762890 2571 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762894 2571 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762899 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762905 2571 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762909 2571 flags.go:64] FLAG: --v="2" Sep 11 14:14:57.771986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762916 2571 flags.go:64] FLAG: --version="false" Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762922 2571 flags.go:64] FLAG: --vmodule="" Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762951 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.762958 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763162 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763173 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763180 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763186 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763192 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763197 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763201 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763207 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763211 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763219 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763225 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763229 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763234 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763238 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763246 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763252 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.772609 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763257 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763261 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763265 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763269 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763274 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763278 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763283 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763288 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763292 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763297 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763301 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763305 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763309 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763313 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763318 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763322 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763326 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763330 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763335 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763339 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.773124 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763343 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763347 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763353 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763357 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763361 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763367 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763371 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763375 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763380 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763384 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763388 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763393 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763397 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763401 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763406 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763410 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763414 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763418 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763422 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763426 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.773979 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763430 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763434 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763438 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763443 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763447 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763451 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763455 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763459 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763463 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763467 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763471 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763475 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763480 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763484 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763489 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763493 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763497 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763503 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763507 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763511 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.774641 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763515 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763519 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763523 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763530 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763537 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763541 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763546 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763550 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763555 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763559 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763564 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763568 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763573 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763577 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763582 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763586 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763590 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763595 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763599 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.775310 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763603 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763607 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763611 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.763616 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.764507 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.772617 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.772635 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.772651 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.772657 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772716 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772721 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772724 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772727 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772730 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.775821 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772735 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772739 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772742 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772747 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772750 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772753 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772755 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772759 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772761 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772764 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772767 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772770 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772772 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772776 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772780 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772782 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772785 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772788 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772790 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772793 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.776191 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772795 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772798 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772801 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772804 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772809 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772812 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772814 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772817 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772820 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772823 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772826 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772828 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772831 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772834 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772836 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772839 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772842 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772844 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772847 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.776683 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772850 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772852 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772855 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772858 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772860 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772863 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772865 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772868 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772870 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772873 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772875 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772878 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772880 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772882 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772885 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772888 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772890 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772893 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772896 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772899 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.777177 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772901 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772904 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772907 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772909 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772912 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772914 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772917 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772921 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772925 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772928 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772947 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772951 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772954 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772957 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772960 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772963 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772965 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772968 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772971 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.777666 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772973 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772976 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772978 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772981 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772984 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772986 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772989 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772992 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772994 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.772997 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773001 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773004 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773007 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773009 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773012 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.778152 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773015 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.773020 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773496 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773732 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773737 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773742 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773747 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773752 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773756 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773769 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773773 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773777 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773782 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773786 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773791 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:57.778520 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773795 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773799 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773804 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773811 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773817 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773822 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773831 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773836 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773841 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773847 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773851 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773856 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773861 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773865 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773869 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773874 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773878 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773883 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773888 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773898 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:57.778892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773902 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773906 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773911 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773915 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773919 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773923 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773927 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773948 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773953 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773958 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773962 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773971 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773975 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773979 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773984 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773988 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773992 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.773996 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774001 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:57.779434 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774005 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774010 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774015 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774020 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774024 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774034 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774039 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774043 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774047 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774052 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774056 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774060 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774064 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774068 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774073 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774077 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774081 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774085 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774093 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774098 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:57.779902 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774105 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774112 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774118 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774124 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774129 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774134 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774138 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774143 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774147 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774152 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774162 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774167 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774172 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774176 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774181 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774186 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774190 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774194 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774199 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774203 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:57.780425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774207 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774211 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774220 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774224 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774229 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774233 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:57.774237 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.774246 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.774673 2571 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.777134 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.778258 2571 server.go:1019] "Starting client certificate rotation" Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.778355 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:57.780913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.779282 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:57.781263 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.781156 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:57.782342 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.782322 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:57.793810 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.793792 2571 log.go:25] "Validated CRI v1 runtime API" Sep 11 14:14:57.798431 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.798413 2571 log.go:25] "Validated CRI v1 image API" Sep 11 14:14:57.799339 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.799320 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 14:14:57.804207 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.804182 2571 fs.go:135] Filesystem UUIDs: map[29fb519f-5256-443a-bfca-3ba00949d12b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d4fb86e4-d173-4b48-9333-fb921dad6a78:/dev/nvme0n1p4] Sep 11 14:14:57.804264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.804208 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 14:14:57.810522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.810394 2571 manager.go:216] Machine: {Timestamp:2026-09-11 14:14:57.808996159 +0000 UTC m=+0.380653639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100896 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2643413b266acb7258a7c639d9549a SystemUUID:ec264341-3b26-6acb-7258-a7c639d9549a BootID:2bcb1c6f-3a1f-47e3-aa1f-1d5936bb9823 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:b9:87:85:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:b9:87:85:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:21:02:bf:b9:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 14:14:57.810522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.810507 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 14:14:57.810661 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.810604 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 14:14:57.813696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.813677 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:57.814128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.814093 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 14:14:57.814303 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.814133 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 14:14:57.815010 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.815000 2571 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 14:14:57.815042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.815013 2571 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 14:14:57.815042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.815021 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:57.815715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.815704 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:57.816964 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.816953 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 14:14:57.821721 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821708 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:57.821858 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821848 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 14:14:57.821891 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821882 2571 kubelet.go:486] "Attempting to sync node with API server" Sep 11 14:14:57.821919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821895 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 14:14:57.821919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821911 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 14:14:57.821999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821921 2571 kubelet.go:398] "Adding apiserver pod source" Sep 11 14:14:57.821999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.821944 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 14:14:57.822471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.822452 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:57.822506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.822490 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:57.826159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.825973 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 14:14:57.829246 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.829231 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 14:14:57.829305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.829262 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 14:14:57.831232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831220 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 14:14:57.831277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831237 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 14:14:57.831277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831244 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 14:14:57.831277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831249 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 14:14:57.831277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831262 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 14:14:57.831277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831270 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831286 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831292 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831299 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831304 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831313 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 14:14:57.831419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.831318 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 14:14:57.832258 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.832248 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 14:14:57.832258 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.832258 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 14:14:57.834260 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.834232 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 14:14:57.834640 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.834337 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 14:14:57.836267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.836255 2571 server.go:1268] "Started kubelet" Sep 11 14:14:57.836425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.836325 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 14:14:57.836425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.836385 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 14:14:57.836553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.836439 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 14:14:57.837146 ip-10-0-141-198 systemd[1]: Started Kubernetes Kubelet. Sep 11 14:14:57.837376 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.837237 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sj4sp" Sep 11 14:14:57.837929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.837911 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 14:14:57.838036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.838024 2571 server.go:310] "Adding debug handlers to kubelet server" Sep 11 14:14:57.844642 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.844625 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 14:14:57.846727 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.846711 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sj4sp" Sep 11 14:14:57.849306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.849290 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 14:14:57.849400 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.849390 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 14:14:57.849461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.849434 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:57.849512 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.849492 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:57.850193 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850164 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 14:14:57.850284 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.849083 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-198.ec2.internal.18d449cd3e5b6668 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-198.ec2.internal,UID:ip-10-0-141-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-198.ec2.internal,},FirstTimestamp:2026-09-11 14:14:57.836230248 +0000 UTC m=+0.407887711,LastTimestamp:2026-09-11 14:14:57.836230248 +0000 UTC m=+0.407887711,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-198.ec2.internal,}" Sep 11 14:14:57.850353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850310 2571 factory.go:55] Registering systemd factory Sep 11 14:14:57.850353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850325 2571 factory.go:223] Registration of the systemd container factory successfully Sep 11 14:14:57.850426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850312 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 14:14:57.850426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850377 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 14:14:57.850489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850443 2571 reconstruct.go:97] "Volume reconstruction finished" Sep 11 14:14:57.850489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850452 2571 reconciler.go:29] "Reconciler: start to sync state" Sep 11 14:14:57.850653 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850641 2571 factory.go:153] Registering CRI-O factory Sep 11 14:14:57.850691 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850658 2571 factory.go:223] Registration of the crio container factory successfully Sep 11 14:14:57.850726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850712 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 14:14:57.850762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850730 2571 factory.go:103] Registering Raw factory Sep 11 14:14:57.850762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.850740 2571 manager.go:1196] Started watching for new ooms in manager Sep 11 14:14:57.851154 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.851139 2571 manager.go:349] Starting recovery of all containers Sep 11 14:14:57.855604 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.855585 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:57.860339 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.860309 2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-198.ec2.internal\" not found" node="ip-10-0-141-198.ec2.internal" Sep 11 14:14:57.863800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.863781 2571 manager.go:354] Recovery completed Sep 11 14:14:57.866048 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.866003 2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 14:14:57.869846 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.869833 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:57.871648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.871629 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:57.871716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.871668 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:57.871716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.871682 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:57.872295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.872282 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 14:14:57.872295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.872294 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 14:14:57.872426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.872301 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:57.872426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.872311 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:57.873014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.872997 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:57.873107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.873025 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:57.873107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.873034 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:57.874476 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.874464 2571 policy_none.go:49] "None policy: Start" Sep 11 14:14:57.874523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.874479 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 14:14:57.874523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.874489 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 14:14:57.876448 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.876435 2571 policy_none.go:47] "Start" Sep 11 14:14:57.911602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.911587 2571 manager.go:337] "Starting Device Plugin manager" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.911623 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.911633 2571 server.go:87] "Starting device plugin registration server" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.911983 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.911995 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.912105 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.912200 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.912209 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.912762 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 14:14:57.932535 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.912816 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:57.950948 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.950905 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 14:14:57.952182 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.952164 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 14:14:57.952239 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.952222 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 14:14:57.952285 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.952253 2571 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 14:14:57.952335 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:57.952298 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 14:14:57.955156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:57.955134 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:58.012240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.012126 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:58.013358 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.013339 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:58.013460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.013370 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:58.013460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.013380 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:58.013460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.013406 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.026516 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.026497 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.026594 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.026524 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-198.ec2.internal\": node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.052652 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.052627 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal"] Sep 11 14:14:58.052771 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.052693 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:58.053132 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.053118 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.053761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.053745 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:58.053831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.053778 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:58.053831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.053793 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:58.056183 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056171 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:58.056299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.056343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056312 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:58.056959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056925 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:58.057056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056971 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:58.057056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056985 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:58.057166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.056986 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:58.057166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.057130 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:58.057166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.057144 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:58.059284 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.059267 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.059371 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.059294 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:58.060132 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.060112 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:58.060226 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.060142 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:58.060226 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.060172 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:58.091655 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.091628 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-198.ec2.internal\" not found" node="ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.095964 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.095946 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-198.ec2.internal\" not found" node="ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.152045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.152017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.152171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.152048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.152171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.152067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2471a11920facda24bd48e50635582ab-config\") pod \"kube-apiserver-proxy-ip-10-0-141-198.ec2.internal\" (UID: \"2471a11920facda24bd48e50635582ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.154129 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.154112 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.252584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252551 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.252715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252585 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.252715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252638 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.252715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2471a11920facda24bd48e50635582ab-config\") pod \"kube-apiserver-proxy-ip-10-0-141-198.ec2.internal\" (UID: \"2471a11920facda24bd48e50635582ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.252715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252681 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2471a11920facda24bd48e50635582ab-config\") pod \"kube-apiserver-proxy-ip-10-0-141-198.ec2.internal\" (UID: \"2471a11920facda24bd48e50635582ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.252863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.252729 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a25abd9a4f516287dba437856f23b137-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal\" (UID: \"a25abd9a4f516287dba437856f23b137\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.254625 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.254608 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.355356 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.355268 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.394461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.394429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.399025 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.399004 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:58.455527 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.455492 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.556233 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.556192 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.657354 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.657274 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.758465 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.758434 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.778481 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.778451 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 14:14:58.778655 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.778629 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:14:58.778699 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.778682 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:14:58.850676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.850629 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:09:58 +0000 UTC" deadline="2028-04-18 00:01:07.949798299 +0000 UTC" Sep 11 14:14:58.850824 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.850674 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14025h46m9.099128839s" Sep 11 14:14:58.859235 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.859211 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.926362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.926335 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:58.960079 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:58.960053 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-198.ec2.internal\" not found" Sep 11 14:14:58.997926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:58.997893 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:59.033864 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:59.033678 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2471a11920facda24bd48e50635582ab.slice/crio-ffa3a22fa31c6e1ba458bdbc282cd4806c12febc081de04186a407b493defb20 WatchSource:0}: Error finding container ffa3a22fa31c6e1ba458bdbc282cd4806c12febc081de04186a407b493defb20: Status 404 returned error can't find the container with id ffa3a22fa31c6e1ba458bdbc282cd4806c12febc081de04186a407b493defb20 Sep 11 14:14:59.040515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.040484 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:14:59.045867 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:14:59.045839 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25abd9a4f516287dba437856f23b137.slice/crio-590a70cc273772c97b36e26c5266026bee9e1f5e54d12f9ff22c3c32a16035ff WatchSource:0}: Error finding container 590a70cc273772c97b36e26c5266026bee9e1f5e54d12f9ff22c3c32a16035ff: Status 404 returned error can't find the container with id 590a70cc273772c97b36e26c5266026bee9e1f5e54d12f9ff22c3c32a16035ff Sep 11 14:14:59.050441 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.050423 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" Sep 11 14:14:59.061746 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.061722 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:59.062698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.062681 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" Sep 11 14:14:59.073934 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.073916 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:59.695355 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.695314 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:59.823010 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.822962 2571 apiserver.go:52] "Watching apiserver" Sep 11 14:14:59.830344 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.830312 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 14:14:59.832419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.832391 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7tmld","kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm","openshift-cluster-node-tuning-operator/tuned-q64r6","openshift-dns/node-resolver-5dl9d","openshift-multus/multus-5x8qs","openshift-network-diagnostics/network-check-target-drq5n","openshift-ovn-kubernetes/ovnkube-node-vbvjq","kube-system/global-pull-secret-syncer-rv7pf","openshift-image-registry/node-ca-n228s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal","openshift-multus/multus-additional-cni-plugins-nxl7l","openshift-multus/network-metrics-daemon-jnjx5","openshift-network-operator/iptables-alerter-fqv7n"] Sep 11 14:14:59.836040 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.835626 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:14:59.836040 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.835710 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:14:59.837597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.837573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.837714 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.837652 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:14:59.839142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.838991 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:59.840081 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.840037 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.842602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.842187 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 14:14:59.842602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.842214 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.842602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.842218 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.842602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.842356 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nwt69\"" Sep 11 14:14:59.842602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.842567 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.844630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.844285 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.844630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.844484 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9m29t\"" Sep 11 14:14:59.844630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.844506 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.844838 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.844816 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.848045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.846507 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npqv2\"" Sep 11 14:14:59.848045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.846954 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.848045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.847173 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.848045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.847341 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.848835 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.849392 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.849597 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m548t\"" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.849824 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.850144 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.851313 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.851377 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:14:59.851856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.851714 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:59.852768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.851994 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.854080 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.853763 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 14:14:59.854080 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.853853 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 14:14:59.854080 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.853913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vkn2t\"" Sep 11 14:14:59.854475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.854454 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:14:59.856203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.856177 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.856549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.856530 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5tx9z\"" Sep 11 14:14:59.856755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.856735 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.856855 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.856769 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 14:14:59.857064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.857046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n228s" Sep 11 14:14:59.859101 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.858940 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.859101 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.858955 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4fxn9\"" Sep 11 14:14:59.859289 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.859180 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 14:14:59.859289 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.859185 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.859583 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.859564 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.861301 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpqg\" (UniqueName: \"kubernetes.io/projected/156459d2-1229-42f4-9f9b-09b858d3b4d2-kube-api-access-jlpqg\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.861412 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-device-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.861412 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.861531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxs95\" (UniqueName: \"kubernetes.io/projected/6f30b28c-23de-4e75-b962-15d050ccff54-kube-api-access-gxs95\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.861531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ad6899-5f03-45f2-8c2e-9099ab141fa8-konnectivity-ca\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.861531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861487 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-var-lib-kubelet\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.861531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhpz\" (UniqueName: \"kubernetes.io/projected/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-kube-api-access-5fhpz\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.861827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861541 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-system-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.861827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861563 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-65s69\"" Sep 11 14:14:59.861827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861711 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 14:14:59.861827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861788 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 14:14:59.861827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861794 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 14:14:59.862142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.861567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-cni-binary-copy\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-socket-dir-parent\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862073 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.862142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862101 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 14:14:59.862360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862247 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 14:14:59.862360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862075 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-socket-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.862360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862347 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 14:14:59.862484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysconfig\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.862484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862398 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-kubernetes\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.862484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-sys\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.862484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862460 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.862668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862491 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-bin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-multus\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862593 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-multus-daemon-config\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.862668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-host\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862686 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be04675b-fc49-48cc-9121-3a49b2ecb6fb-tmp-dir\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862710 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-k8s-cni-cncf-io\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-multus-certs\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.862897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-conf\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ad6899-5f03-45f2-8c2e-9099ab141fa8-agent-certs\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.862991 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-systemd\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-lib-modules\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863088 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-tuned\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863131 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-cnibin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-registration-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.863195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-sys-fs\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863219 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-tmp\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be04675b-fc49-48cc-9121-3a49b2ecb6fb-hosts-file\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863275 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-kubelet\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-hostroot\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863367 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-etc-kubernetes\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-kubelet-config\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xs2g\" (UniqueName: \"kubernetes.io/projected/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kube-api-access-4xs2g\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863467 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-modprobe-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863492 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-run\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f7x\" (UniqueName: \"kubernetes.io/projected/be04675b-fc49-48cc-9121-3a49b2ecb6fb-kube-api-access-r4f7x\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863544 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-netns\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-conf-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.863614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-dbus\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.864084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.863644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:14:59.864201 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.864181 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 14:14:59.864258 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.864247 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnk2c\"" Sep 11 14:14:59.864372 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.864183 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 14:14:59.866748 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.866718 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:59.951260 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.951183 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 14:14:59.957952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.957890 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" event={"ID":"a25abd9a4f516287dba437856f23b137","Type":"ContainerStarted","Data":"590a70cc273772c97b36e26c5266026bee9e1f5e54d12f9ff22c3c32a16035ff"} Sep 11 14:14:59.959472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.959443 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" event={"ID":"2471a11920facda24bd48e50635582ab","Type":"ContainerStarted","Data":"ffa3a22fa31c6e1ba458bdbc282cd4806c12febc081de04186a407b493defb20"} Sep 11 14:14:59.963627 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.963602 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2l7hn" Sep 11 14:14:59.964232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7t2z\" (UniqueName: \"kubernetes.io/projected/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-kube-api-access-j7t2z\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-cnibin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-registration-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964289 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-tuned\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-kubelet\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-hostroot\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964373 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-cnibin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-kubelet-config\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.964722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964456 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-kubelet-config\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.964722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964506 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-kubelet\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-hostroot\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964587 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-registration-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xs2g\" (UniqueName: \"kubernetes.io/projected/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kube-api-access-4xs2g\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964752 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-host\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-conf-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964844 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-dbus\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964867 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964896 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpqg\" (UniqueName: \"kubernetes.io/projected/156459d2-1229-42f4-9f9b-09b858d3b4d2-kube-api-access-jlpqg\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-device-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.964961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964945 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ad6899-5f03-45f2-8c2e-9099ab141fa8-konnectivity-ca\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964978 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-netns\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.964996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-log-socket\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965017 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxs95\" (UniqueName: \"kubernetes.io/projected/6f30b28c-23de-4e75-b962-15d050ccff54-kube-api-access-gxs95\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhpz\" (UniqueName: \"kubernetes.io/projected/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-kube-api-access-5fhpz\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-node-log\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965093 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28d310f-ccc4-440c-8265-5e9147908c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965145 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 14:14:59.965452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965438 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-conf-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.965933 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965537 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f74ee825-31fc-4842-8b9f-257b99bab445-dbus\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.965933 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965665 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966045 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-device-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.966158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.965152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-script-lib\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966117 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ad6899-5f03-45f2-8c2e-9099ab141fa8-konnectivity-ca\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.966261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966150 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-system-cni-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.966261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cnibin\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.966261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-socket-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.966261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysconfig\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-kubernetes\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc26373a-dc49-4ce5-b93b-bcf96e419b92-iptables-alerter-script\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966338 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-serviceca\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysconfig\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966354 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-socket-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-multus\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-multus-daemon-config\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.966473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-multus\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-host\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-slash\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966549 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxf4\" (UniqueName: \"kubernetes.io/projected/b28d310f-ccc4-440c-8265-5e9147908c6b-kube-api-access-kdxf4\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966588 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-host\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-k8s-cni-cncf-io\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966621 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-k8s-cni-cncf-io\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966631 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966490 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-kubernetes\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-conf\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-bin\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.966730 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-netd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ad6899-5f03-45f2-8c2e-9099ab141fa8-agent-certs\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.966993 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.966779 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.466762844 +0000 UTC m=+3.038420297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.966944 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-multus-daemon-config\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-ovn\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967024 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-conf\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967062 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-sys-fs\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967108 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-sys-fs\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-tmp\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-config\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc26373a-dc49-4ce5-b93b-bcf96e419b92-host-slash\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-lib-modules\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be04675b-fc49-48cc-9121-3a49b2ecb6fb-hosts-file\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-etc-kubernetes\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-modprobe-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-run\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967404 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-lib-modules\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-sys\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967447 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-var-lib-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967464 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be04675b-fc49-48cc-9121-3a49b2ecb6fb-hosts-file\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.967947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f7x\" (UniqueName: \"kubernetes.io/projected/be04675b-fc49-48cc-9121-3a49b2ecb6fb-kube-api-access-r4f7x\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967500 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-etc-kubernetes\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-netns\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967532 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-systemd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967542 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-sys\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pjp\" (UniqueName: \"kubernetes.io/projected/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-kube-api-access-v9pjp\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967589 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-var-lib-kubelet\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967613 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-modprobe-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-etc-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-env-overrides\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-system-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-cni-binary-copy\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967729 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-socket-dir-parent\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967775 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-bin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be04675b-fc49-48cc-9121-3a49b2ecb6fb-tmp-dir\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967824 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-multus-certs\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.968822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-systemd-units\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967880 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-netns\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnkj\" (UniqueName: \"kubernetes.io/projected/cc26373a-dc49-4ce5-b93b-bcf96e419b92-kube-api-access-zcnkj\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967958 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-var-lib-kubelet\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.967964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-systemd\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968018 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-kubelet\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968005 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-var-lib-cni-bin\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968159 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-run\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968215 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-system-cni-dir\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968312 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be04675b-fc49-48cc-9121-3a49b2ecb6fb-tmp-dir\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.968455 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968468 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-sysctl-d\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.968512 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.46849508 +0000 UTC m=+3.040152539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968555 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b94420c-3fde-42a6-84bd-2c9afa4b6661-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.969496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968574 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-multus-socket-dir-parent\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.970347 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968611 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-systemd\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.970347 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968623 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f30b28c-23de-4e75-b962-15d050ccff54-host-run-multus-certs\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.970347 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.968706 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f30b28c-23de-4e75-b962-15d050ccff54-cni-binary-copy\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.970347 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.970328 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-etc-tuned\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.971031 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.970867 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ad6899-5f03-45f2-8c2e-9099ab141fa8-agent-certs\") pod \"konnectivity-agent-7tmld\" (UID: \"04ad6899-5f03-45f2-8c2e-9099ab141fa8\") " pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:14:59.971185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.971165 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-tmp\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.974809 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.974781 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xs2g\" (UniqueName: \"kubernetes.io/projected/9b94420c-3fde-42a6-84bd-2c9afa4b6661-kube-api-access-4xs2g\") pod \"aws-ebs-csi-driver-node-kc8nm\" (UID: \"9b94420c-3fde-42a6-84bd-2c9afa4b6661\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:14:59.974914 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.974884 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpqg\" (UniqueName: \"kubernetes.io/projected/156459d2-1229-42f4-9f9b-09b858d3b4d2-kube-api-access-jlpqg\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:14:59.976240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.976219 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2l7hn" Sep 11 14:14:59.977289 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.977267 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:59.977387 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.977294 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:59.977387 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.977310 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:59.977387 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:14:59.977370 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.477352155 +0000 UTC m=+3.049009612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:59.985024 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.984964 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxs95\" (UniqueName: \"kubernetes.io/projected/6f30b28c-23de-4e75-b962-15d050ccff54-kube-api-access-gxs95\") pod \"multus-5x8qs\" (UID: \"6f30b28c-23de-4e75-b962-15d050ccff54\") " pod="openshift-multus/multus-5x8qs" Sep 11 14:14:59.985481 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.985454 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhpz\" (UniqueName: \"kubernetes.io/projected/56e75d0e-d340-4f76-a0fc-9389cba0f1a9-kube-api-access-5fhpz\") pod \"tuned-q64r6\" (UID: \"56e75d0e-d340-4f76-a0fc-9389cba0f1a9\") " pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:14:59.994778 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:14:59.994716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f7x\" (UniqueName: \"kubernetes.io/projected/be04675b-fc49-48cc-9121-3a49b2ecb6fb-kube-api-access-r4f7x\") pod \"node-resolver-5dl9d\" (UID: \"be04675b-fc49-48cc-9121-3a49b2ecb6fb\") " pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:15:00.068833 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-script-lib\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.068833 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-system-cni-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068871 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cnibin\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc26373a-dc49-4ce5-b93b-bcf96e419b92-iptables-alerter-script\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068970 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cnibin\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.068970 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-system-cni-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069047 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-serviceca\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.069127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-slash\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxf4\" (UniqueName: \"kubernetes.io/projected/b28d310f-ccc4-440c-8265-5e9147908c6b-kube-api-access-kdxf4\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069245 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-bin\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069293 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-bin\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-netd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-ovn\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-config\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069380 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc26373a-dc49-4ce5-b93b-bcf96e419b92-host-slash\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-var-lib-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-systemd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.069488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pjp\" (UniqueName: \"kubernetes.io/projected/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-kube-api-access-v9pjp\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069498 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc26373a-dc49-4ce5-b93b-bcf96e419b92-iptables-alerter-script\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-etc-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-env-overrides\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-etc-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-systemd-units\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069590 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc26373a-dc49-4ce5-b93b-bcf96e419b92-host-slash\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069600 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnkj\" (UniqueName: \"kubernetes.io/projected/cc26373a-dc49-4ce5-b93b-bcf96e419b92-kube-api-access-zcnkj\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069628 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-serviceca\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-kubelet\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069654 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-cni-netd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-kubelet\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069627 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-script-lib\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069678 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-systemd-units\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069704 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-ovn\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.070161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069708 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7t2z\" (UniqueName: \"kubernetes.io/projected/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-kube-api-access-j7t2z\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-host\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069827 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069867 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-netns\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069875 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-log-socket\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069933 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-node-log\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069951 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-run-systemd\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28d310f-ccc4-440c-8265-5e9147908c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070057 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-var-lib-openvswitch\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.069209 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-slash\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070100 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070115 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-run-netns\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070142 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070164 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-ovnkube-config\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070188 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-log-socket\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070216 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-host\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.071889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070227 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28d310f-ccc4-440c-8265-5e9147908c6b-node-log\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070326 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28d310f-ccc4-440c-8265-5e9147908c6b-env-overrides\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.071889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.070570 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.072690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.072667 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28d310f-ccc4-440c-8265-5e9147908c6b-ovn-node-metrics-cert\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.080310 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.080282 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxf4\" (UniqueName: \"kubernetes.io/projected/b28d310f-ccc4-440c-8265-5e9147908c6b-kube-api-access-kdxf4\") pod \"ovnkube-node-vbvjq\" (UID: \"b28d310f-ccc4-440c-8265-5e9147908c6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.080576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.080553 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pjp\" (UniqueName: \"kubernetes.io/projected/764a654a-de8b-4e7d-9bd6-029cc4d6f79c-kube-api-access-v9pjp\") pod \"multus-additional-cni-plugins-nxl7l\" (UID: \"764a654a-de8b-4e7d-9bd6-029cc4d6f79c\") " pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.080678 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.080660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7t2z\" (UniqueName: \"kubernetes.io/projected/b30ec984-eb09-4f9e-aa21-d12eaa0e73ad-kube-api-access-j7t2z\") pod \"node-ca-n228s\" (UID: \"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad\") " pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.080981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.080962 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnkj\" (UniqueName: \"kubernetes.io/projected/cc26373a-dc49-4ce5-b93b-bcf96e419b92-kube-api-access-zcnkj\") pod \"iptables-alerter-fqv7n\" (UID: \"cc26373a-dc49-4ce5-b93b-bcf96e419b92\") " pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.157429 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.157394 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" Sep 11 14:15:00.165391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.165363 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q64r6" Sep 11 14:15:00.184379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.184344 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5dl9d" Sep 11 14:15:00.191382 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.191358 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5x8qs" Sep 11 14:15:00.200275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.200254 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:15:00.208111 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.208081 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqv7n" Sep 11 14:15:00.217055 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.217034 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n228s" Sep 11 14:15:00.225112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.225089 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:00.231103 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.231081 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" Sep 11 14:15:00.473178 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.473086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:00.473353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.473175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:00.473353 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.473230 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:00.473353 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.473271 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:00.473353 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.473317 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:01.473294222 +0000 UTC m=+4.044951693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:00.473353 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.473337 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:01.473327731 +0000 UTC m=+4.044985183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:00.573625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.573590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:00.573826 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.573801 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:00.573902 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.573829 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:00.573902 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.573839 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:00.574054 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:00.573910 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:01.573889606 +0000 UTC m=+4.145547077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:00.980917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.980879 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:10:00 +0000 UTC" deadline="2028-03-06 08:39:59.69429581 +0000 UTC" Sep 11 14:15:00.981316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:00.980914 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13002h24m58.713386866s" Sep 11 14:15:01.030156 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.030128 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe04675b_fc49_48cc_9121_3a49b2ecb6fb.slice/crio-7d1291ee44997cecabf72fd257e7c8f005336ef4eacebfd56cb9651e2d106d64 WatchSource:0}: Error finding container 7d1291ee44997cecabf72fd257e7c8f005336ef4eacebfd56cb9651e2d106d64: Status 404 returned error can't find the container with id 7d1291ee44997cecabf72fd257e7c8f005336ef4eacebfd56cb9651e2d106d64 Sep 11 14:15:01.030697 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.030670 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e75d0e_d340_4f76_a0fc_9389cba0f1a9.slice/crio-d6e18f84ddaaa3cd60d9587ad1acb62b64c7e960f07c5fd0c90bcdd7a96a08a3 WatchSource:0}: Error finding container d6e18f84ddaaa3cd60d9587ad1acb62b64c7e960f07c5fd0c90bcdd7a96a08a3: Status 404 returned error can't find the container with id d6e18f84ddaaa3cd60d9587ad1acb62b64c7e960f07c5fd0c90bcdd7a96a08a3 Sep 11 14:15:01.032408 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.032386 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc26373a_dc49_4ce5_b93b_bcf96e419b92.slice/crio-c15a0ff78a2184176dc41dd5722843b1e48c53af64cdd425a459d0709d233a58 WatchSource:0}: Error finding container c15a0ff78a2184176dc41dd5722843b1e48c53af64cdd425a459d0709d233a58: Status 404 returned error can't find the container with id c15a0ff78a2184176dc41dd5722843b1e48c53af64cdd425a459d0709d233a58 Sep 11 14:15:01.035735 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.035712 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b94420c_3fde_42a6_84bd_2c9afa4b6661.slice/crio-4144778912dd4ee652a88ed6dce2fb8b44b96d8d281ec2f63c6702194285c1fd WatchSource:0}: Error finding container 4144778912dd4ee652a88ed6dce2fb8b44b96d8d281ec2f63c6702194285c1fd: Status 404 returned error can't find the container with id 4144778912dd4ee652a88ed6dce2fb8b44b96d8d281ec2f63c6702194285c1fd Sep 11 14:15:01.038260 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.037894 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ad6899_5f03_45f2_8c2e_9099ab141fa8.slice/crio-b53dcdc67da4f325ca281a31b892959a287d4dbc4ba63dc90083c1ba4558edce WatchSource:0}: Error finding container b53dcdc67da4f325ca281a31b892959a287d4dbc4ba63dc90083c1ba4558edce: Status 404 returned error can't find the container with id b53dcdc67da4f325ca281a31b892959a287d4dbc4ba63dc90083c1ba4558edce Sep 11 14:15:01.038503 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.038399 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f30b28c_23de_4e75_b962_15d050ccff54.slice/crio-87451b25e2390f54be9be786d94850f1d1f31089e3721305ce1781d6df49d812 WatchSource:0}: Error finding container 87451b25e2390f54be9be786d94850f1d1f31089e3721305ce1781d6df49d812: Status 404 returned error can't find the container with id 87451b25e2390f54be9be786d94850f1d1f31089e3721305ce1781d6df49d812 Sep 11 14:15:01.039940 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.039631 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764a654a_de8b_4e7d_9bd6_029cc4d6f79c.slice/crio-377eb905d7949c617d5c19ec8d6e7fc778e89a50d7ca08973ee59313e68c2587 WatchSource:0}: Error finding container 377eb905d7949c617d5c19ec8d6e7fc778e89a50d7ca08973ee59313e68c2587: Status 404 returned error can't find the container with id 377eb905d7949c617d5c19ec8d6e7fc778e89a50d7ca08973ee59313e68c2587 Sep 11 14:15:01.041451 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:01.041426 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30ec984_eb09_4f9e_aa21_d12eaa0e73ad.slice/crio-38c5790ebfb095e98793a3f246fb5093659828d0768e5e5658be9bd2ab29f961 WatchSource:0}: Error finding container 38c5790ebfb095e98793a3f246fb5093659828d0768e5e5658be9bd2ab29f961: Status 404 returned error can't find the container with id 38c5790ebfb095e98793a3f246fb5093659828d0768e5e5658be9bd2ab29f961 Sep 11 14:15:01.480843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.480808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:01.480843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.480854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:01.481080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.480950 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:01.481080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.480974 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:01.481080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.481025 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:03.481011181 +0000 UTC m=+6.052668633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:01.481080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.481040 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:03.481032801 +0000 UTC m=+6.052690254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:01.581804 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.581712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:01.581980 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.581932 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:01.582033 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.581984 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:01.582033 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.582002 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:01.582116 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.582077 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:03.582055485 +0000 UTC m=+6.153712941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.955318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.955463 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.955902 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.956044 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.956116 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:01.956300 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:01.956193 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:01.965282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.965225 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerStarted","Data":"377eb905d7949c617d5c19ec8d6e7fc778e89a50d7ca08973ee59313e68c2587"} Sep 11 14:15:01.970085 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.969979 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" event={"ID":"9b94420c-3fde-42a6-84bd-2c9afa4b6661","Type":"ContainerStarted","Data":"4144778912dd4ee652a88ed6dce2fb8b44b96d8d281ec2f63c6702194285c1fd"} Sep 11 14:15:01.974324 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.974242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqv7n" event={"ID":"cc26373a-dc49-4ce5-b93b-bcf96e419b92","Type":"ContainerStarted","Data":"c15a0ff78a2184176dc41dd5722843b1e48c53af64cdd425a459d0709d233a58"} Sep 11 14:15:01.977149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.977094 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5dl9d" event={"ID":"be04675b-fc49-48cc-9121-3a49b2ecb6fb","Type":"ContainerStarted","Data":"7d1291ee44997cecabf72fd257e7c8f005336ef4eacebfd56cb9651e2d106d64"} Sep 11 14:15:01.981304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.981240 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:10:00 +0000 UTC" deadline="2028-03-06 16:47:40.359459199 +0000 UTC" Sep 11 14:15:01.981304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.981270 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13010h32m38.378193547s" Sep 11 14:15:01.982518 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.982405 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"c16485bf25b1d91441335436556d071a2b8b1a13b1e0e6497d8a26eb7e082e3a"} Sep 11 14:15:01.987594 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.987569 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5x8qs" event={"ID":"6f30b28c-23de-4e75-b962-15d050ccff54","Type":"ContainerStarted","Data":"87451b25e2390f54be9be786d94850f1d1f31089e3721305ce1781d6df49d812"} Sep 11 14:15:01.989421 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.989397 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tmld" event={"ID":"04ad6899-5f03-45f2-8c2e-9099ab141fa8","Type":"ContainerStarted","Data":"b53dcdc67da4f325ca281a31b892959a287d4dbc4ba63dc90083c1ba4558edce"} Sep 11 14:15:01.998298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:01.998266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q64r6" event={"ID":"56e75d0e-d340-4f76-a0fc-9389cba0f1a9","Type":"ContainerStarted","Data":"d6e18f84ddaaa3cd60d9587ad1acb62b64c7e960f07c5fd0c90bcdd7a96a08a3"} Sep 11 14:15:02.001352 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:02.001325 2571 generic.go:356] "Generic (PLEG): container finished" podID="a25abd9a4f516287dba437856f23b137" containerID="ba1a2c89b66c431dd596f733cb2e4b0d3b3954dae5c3d346dbd6059b37402d25" exitCode=0 Sep 11 14:15:02.002014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:02.001989 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" event={"ID":"a25abd9a4f516287dba437856f23b137","Type":"ContainerDied","Data":"ba1a2c89b66c431dd596f733cb2e4b0d3b3954dae5c3d346dbd6059b37402d25"} Sep 11 14:15:02.009361 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:02.009332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" event={"ID":"2471a11920facda24bd48e50635582ab","Type":"ContainerStarted","Data":"bcc36ba381d2b83ef39d9f3dc3f2d6e208889a1acb2bb208aae5928097e6f2fb"} Sep 11 14:15:02.015133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:02.012205 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n228s" event={"ID":"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad","Type":"ContainerStarted","Data":"38c5790ebfb095e98793a3f246fb5093659828d0768e5e5658be9bd2ab29f961"} Sep 11 14:15:02.057215 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:02.057142 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-198.ec2.internal" podStartSLOduration=2.057115673 podStartE2EDuration="2.057115673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:02.047202158 +0000 UTC m=+4.618859638" watchObservedRunningTime="2026-09-11 14:15:02.057115673 +0000 UTC m=+4.628773148" Sep 11 14:15:03.029749 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.029476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" event={"ID":"a25abd9a4f516287dba437856f23b137","Type":"ContainerStarted","Data":"4b8db199e429b0dec08e3717ab11e4841b8cddf365a4c6d8beffbaae74d25a1b"} Sep 11 14:15:03.498517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.498419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:03.498695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.498528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:03.498695 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.498679 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:03.498810 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.498744 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:07.498725791 +0000 UTC m=+10.070383249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:03.499196 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.499174 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:03.499286 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.499233 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:07.499218328 +0000 UTC m=+10.070875786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:03.600439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.599671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:03.600439 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.599916 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:03.600439 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.599939 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:03.600439 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.599956 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:03.600439 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.600040 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:07.600018215 +0000 UTC m=+10.171675669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.953087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.953226 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.953634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.953731 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:03.953799 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:03.953984 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:03.953886 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:05.953585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:05.953546 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:05.954101 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:05.953705 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:05.954101 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:05.953910 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:05.954101 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:05.954058 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:05.954101 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:05.954085 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:05.954323 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:05.954211 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:07.538203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.538149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:07.538667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.538228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:07.538667 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.538362 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:07.538667 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.538430 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.538408804 +0000 UTC m=+18.110066279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:07.538667 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.538497 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:07.538667 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.538534 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.538520654 +0000 UTC m=+18.110178124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:07.639556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.639477 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:07.639761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.639649 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:07.639761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.639670 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:07.639761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.639683 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:07.639761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.639744 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.639725534 +0000 UTC m=+18.211382994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:07.954864 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.954774 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:07.955064 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.954893 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:07.955351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.955328 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:07.955475 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.955453 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:07.955646 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:07.955606 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:07.955716 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:07.955697 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:09.953328 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:09.953285 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:09.953830 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:09.953420 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:09.953830 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:09.953794 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:09.953954 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:09.953912 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:09.954036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:09.953981 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:09.954179 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:09.954132 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:11.953237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:11.953199 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:11.953237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:11.953225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:11.953761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:11.953199 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:11.953761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:11.953307 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:11.953761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:11.953399 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:11.953761 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:11.953486 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:13.952708 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:13.952652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:13.953171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:13.952661 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:13.953171 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:13.952788 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:13.953171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:13.952794 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:13.953171 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:13.952893 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:13.953171 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:13.953046 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:15.604106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.604079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:15.604573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.604125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:15.604573 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.604225 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:15.604573 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.604249 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:15.604573 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.604297 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.604276294 +0000 UTC m=+34.175933765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:15.604573 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.604316 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.604306448 +0000 UTC m=+34.175963898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:15.704838 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.704800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:15.705044 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.704994 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:15.705044 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.705018 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:15.705044 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.705032 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:15.705183 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.705091 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.705070933 +0000 UTC m=+34.276728403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:15.952767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.952684 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:15.952767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.952700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:15.952767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:15.952732 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:15.953064 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.952827 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:15.953064 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.952868 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:15.953064 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:15.952988 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:17.954089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:17.954055 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:17.954561 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:17.954154 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:17.954561 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:17.954163 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:17.954561 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:17.954242 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:17.954561 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:17.954268 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:17.954561 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:17.954341 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:19.952976 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:19.952927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:19.953519 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:19.953069 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:19.953519 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:19.953076 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:19.953519 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:19.953184 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:19.953519 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:19.953239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:19.953519 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:19.953350 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:21.076380 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.076255 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" event={"ID":"9b94420c-3fde-42a6-84bd-2c9afa4b6661","Type":"ContainerStarted","Data":"c88d80e8cfc53ef12debfcba5827288d247d6836d68b7ea8c63d5df8163a9a7b"} Sep 11 14:15:21.078064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.077968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5dl9d" event={"ID":"be04675b-fc49-48cc-9121-3a49b2ecb6fb","Type":"ContainerStarted","Data":"d432179c61643e311cf516d5d4d3778a2afdb00e96165599966ff5aab8c53696"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.081917 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"2fe8ffc0760ade9d1a300566c0e3869f0c2a39826ba8d9325c6e3109c62bddf6"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.082006 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"740215776fbdf64808d7d0388fb76b93ddb766696898b3ae56073b03f7723a1c"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.082043 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"7554ec41b1075f0f58987cf9818cf82bd62ecaa1bf0e80298513e29f27d7c366"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.082089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"6663202b818744eb682a2a5fa7b9e3f5fa76c92c46a267fc1e2cacbff531a0b9"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.082123 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"4ad12c8b7d80e3773606b672b02c0c686760a4fcd03813d02b74b0acbe27b396"} Sep 11 14:15:21.082184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.082149 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"30eb3f42482e347c8862737fbaf5ff05e176b4c5fbc1f6e96f46fbdb7b1b2acd"} Sep 11 14:15:21.083755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.083731 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5x8qs" event={"ID":"6f30b28c-23de-4e75-b962-15d050ccff54","Type":"ContainerStarted","Data":"3cf68ed636684c69f169a29946ff2207b2f7aaff27260efc63726ada57d8e667"} Sep 11 14:15:21.085400 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.085378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7tmld" event={"ID":"04ad6899-5f03-45f2-8c2e-9099ab141fa8","Type":"ContainerStarted","Data":"25e318bee0a3202198d6346e24ab8d76aa1849f70c0c848e54199821ce1e1d6e"} Sep 11 14:15:21.087056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.087033 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q64r6" event={"ID":"56e75d0e-d340-4f76-a0fc-9389cba0f1a9","Type":"ContainerStarted","Data":"277a67da2c56bbd34221fcfdc75df44e3bf7c1601886ad047e06f61200a4f1ee"} Sep 11 14:15:21.088659 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.088637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n228s" event={"ID":"b30ec984-eb09-4f9e-aa21-d12eaa0e73ad","Type":"ContainerStarted","Data":"2f091a5d81afe28c09a49d24da8de6b1847e1b76bdf996c454bc1dae015ab8c7"} Sep 11 14:15:21.090458 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.090437 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="d7bfba21d42891d60c334f9f81d5aea8d908a1d88cb794bd7e1be6d0f8098f64" exitCode=0 Sep 11 14:15:21.090561 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.090469 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"d7bfba21d42891d60c334f9f81d5aea8d908a1d88cb794bd7e1be6d0f8098f64"} Sep 11 14:15:21.096425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.096386 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-198.ec2.internal" podStartSLOduration=21.096353368 podStartE2EDuration="21.096353368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:03.050390949 +0000 UTC m=+5.622048427" watchObservedRunningTime="2026-09-11 14:15:21.096353368 +0000 UTC m=+23.668010836" Sep 11 14:15:21.097667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.097626 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5dl9d" podStartSLOduration=2.911573295 podStartE2EDuration="22.09761251s" totalImagesPullingTime="19.186039215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:21.096255924 +0000 UTC m=+23.667913409" watchObservedRunningTime="2026-09-11 14:15:21.09761251 +0000 UTC m=+23.669269973" Sep 11 14:15:21.120715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.120630 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5x8qs" podStartSLOduration=2.890409588 podStartE2EDuration="22.120609818s" totalImagesPullingTime="19.23020023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:21.120555142 +0000 UTC m=+23.692212638" watchObservedRunningTime="2026-09-11 14:15:21.120609818 +0000 UTC m=+23.692267304" Sep 11 14:15:21.140214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.140164 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q64r6" podStartSLOduration=2.954280914 podStartE2EDuration="22.140145868s" totalImagesPullingTime="19.185864954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:21.139507359 +0000 UTC m=+23.711164858" watchObservedRunningTime="2026-09-11 14:15:21.140145868 +0000 UTC m=+23.711803356" Sep 11 14:15:21.155321 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.155259 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7tmld" podStartSLOduration=2.97633166 podStartE2EDuration="22.15524423s" totalImagesPullingTime="19.17891257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:21.155014108 +0000 UTC m=+23.726671605" watchObservedRunningTime="2026-09-11 14:15:21.15524423 +0000 UTC m=+23.726901703" Sep 11 14:15:21.174498 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.174458 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n228s" podStartSLOduration=2.999663678 podStartE2EDuration="22.174443632s" totalImagesPullingTime="19.174779954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:21.174076225 +0000 UTC m=+23.745733719" watchObservedRunningTime="2026-09-11 14:15:21.174443632 +0000 UTC m=+23.746101313" Sep 11 14:15:21.556960 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.556920 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 14:15:21.923467 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.923310 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:15:21.556954511Z","UUID":"5d8ad2b8-4eea-4080-b562-6573d3fd5c8e","Handler":null,"Name":"","Endpoint":""} Sep 11 14:15:21.925235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.925201 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 14:15:21.925235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.925239 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 14:15:21.953124 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.953082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:21.953124 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.953101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:21.953348 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:21.953215 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:21.953348 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:21.953289 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:21.953348 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:21.953346 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:21.953575 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:21.953417 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:22.095488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:22.095450 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" event={"ID":"9b94420c-3fde-42a6-84bd-2c9afa4b6661","Type":"ContainerStarted","Data":"0661978ed4d92bdbcd97deee5e47aec415ee185a5c00d09fb2a9459b203d4f9a"} Sep 11 14:15:22.097253 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:22.097224 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqv7n" event={"ID":"cc26373a-dc49-4ce5-b93b-bcf96e419b92","Type":"ContainerStarted","Data":"3ed16a73e1fa22e49cef50232e633ea243e12fc77db6092e0d73db4b3a9b1f1b"} Sep 11 14:15:22.114393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:22.114349 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fqv7n" podStartSLOduration=3.929803385 podStartE2EDuration="23.114329321s" totalImagesPullingTime="19.184525936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:22.114062303 +0000 UTC m=+24.685719802" watchObservedRunningTime="2026-09-11 14:15:22.114329321 +0000 UTC m=+24.685986795" Sep 11 14:15:23.101909 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.101858 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" event={"ID":"9b94420c-3fde-42a6-84bd-2c9afa4b6661","Type":"ContainerStarted","Data":"9b3ff338ca7819667e332d350a0d5858d16f3a7c1001b4d4dcb2b4f70fda7c6f"} Sep 11 14:15:23.105732 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.105697 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"52e662bc891f1c38425091888b18b1643f5db5f0bd30e9d63bd7a1d71b56771d"} Sep 11 14:15:23.123882 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.123784 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc8nm" podStartSLOduration=2.9220786739999998 podStartE2EDuration="24.123763793s" totalImagesPullingTime="21.201685119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:23.123470898 +0000 UTC m=+25.695128389" watchObservedRunningTime="2026-09-11 14:15:23.123763793 +0000 UTC m=+25.695421268" Sep 11 14:15:23.953338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.953304 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:23.953524 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.953304 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:23.953524 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:23.953442 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:23.953524 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:23.953304 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:23.953524 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:23.953511 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:23.953683 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:23.953552 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:24.185745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:24.185713 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:15:24.186499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:24.186474 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:15:25.110397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:25.110364 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:15:25.110853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:25.110832 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7tmld" Sep 11 14:15:25.953421 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:25.953323 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:25.953421 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:25.953339 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:25.954168 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:25.953449 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:25.954168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:25.953460 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:25.954168 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:25.953538 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:25.954168 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:25.953612 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:26.115465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.115431 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" event={"ID":"b28d310f-ccc4-440c-8265-5e9147908c6b","Type":"ContainerStarted","Data":"5e2247903013612d69b3b6670ac48daba7c1b50d159560b0142c51f83559445e"} Sep 11 14:15:26.115690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.115672 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:26.115779 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.115736 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:26.117301 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.117278 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="ee2cc78ee39fac3ce9f90c42df37d4bb7dfcf316cddbc41ac838d44deb555998" exitCode=0 Sep 11 14:15:26.117443 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.117364 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"ee2cc78ee39fac3ce9f90c42df37d4bb7dfcf316cddbc41ac838d44deb555998"} Sep 11 14:15:26.130829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.130809 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:26.150808 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:26.150766 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" podStartSLOduration=7.882512209 podStartE2EDuration="27.150753046s" totalImagesPullingTime="19.268240837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:26.150395091 +0000 UTC m=+28.722052579" watchObservedRunningTime="2026-09-11 14:15:26.150753046 +0000 UTC m=+28.722410536" Sep 11 14:15:27.143045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.142534 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:27.162029 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.162001 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:27.286395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.286367 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-drq5n"] Sep 11 14:15:27.286819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.286536 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:27.286819 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:27.286645 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:27.289776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.289739 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv7pf"] Sep 11 14:15:27.289928 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.289845 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:27.290016 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:27.289946 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:27.290341 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.290319 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnjx5"] Sep 11 14:15:27.290417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:27.290400 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:27.290508 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:27.290487 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:28.124015 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:28.123979 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="f54a348f6e2cfaf5259c5d2074a2aac85dcad3578f90a1f16132b62e5892b906" exitCode=0 Sep 11 14:15:28.124175 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:28.124057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"f54a348f6e2cfaf5259c5d2074a2aac85dcad3578f90a1f16132b62e5892b906"} Sep 11 14:15:28.952439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:28.952407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:28.952439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:28.952432 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:28.952976 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:28.952407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:28.952976 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:28.952526 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:28.952976 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:28.952644 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:28.952976 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:28.952738 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:30.130796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:30.130613 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="4ff0b4e133adb15984570a5be459089d921167a4ba4a9578c82e67171643e366" exitCode=0 Sep 11 14:15:30.130796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:30.130698 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"4ff0b4e133adb15984570a5be459089d921167a4ba4a9578c82e67171643e366"} Sep 11 14:15:30.953383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:30.953348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:30.953383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:30.953376 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:30.953668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:30.953376 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:30.953668 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:30.953483 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:30.953668 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:30.953552 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:30.953668 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:30.953644 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:31.619042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:31.619008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:31.619465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:31.619056 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:31.619465 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.619161 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:31.619465 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.619164 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:31.619465 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.619225 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.619209501 +0000 UTC m=+66.190866974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:31.619465 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.619241 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret podName:f74ee825-31fc-4842-8b9f-257b99bab445 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.619235037 +0000 UTC m=+66.190892487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret") pod "global-pull-secret-syncer-rv7pf" (UID: "f74ee825-31fc-4842-8b9f-257b99bab445") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:31.720125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:31.720090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:31.720413 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.720240 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:31.720413 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.720268 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:31.720413 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.720279 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vbwq4 for pod openshift-network-diagnostics/network-check-target-drq5n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:31.720413 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:31.720339 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4 podName:d54aea61-86a8-436d-9665-0820ffd97b55 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.720316641 +0000 UTC m=+66.291974106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwq4" (UniqueName: "kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4") pod "network-check-target-drq5n" (UID: "d54aea61-86a8-436d-9665-0820ffd97b55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:32.516407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:32.516372 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:32.952898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:32.952868 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:32.952898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:32.952895 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:32.953489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:32.952868 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:32.953489 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:32.953001 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-drq5n" podUID="d54aea61-86a8-436d-9665-0820ffd97b55" Sep 11 14:15:32.953489 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:32.953152 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnjx5" podUID="156459d2-1229-42f4-9f9b-09b858d3b4d2" Sep 11 14:15:32.953489 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:32.953255 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv7pf" podUID="f74ee825-31fc-4842-8b9f-257b99bab445" Sep 11 14:15:33.272612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.272581 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-198.ec2.internal" event="NodeReady" Sep 11 14:15:33.272794 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.272709 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 14:15:33.340585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.340040 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rbkk7"] Sep 11 14:15:33.345014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.344993 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qv6jl"] Sep 11 14:15:33.345162 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.345118 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.347183 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.347160 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\"" Sep 11 14:15:33.347335 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.347317 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 14:15:33.347407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.347322 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 14:15:33.348542 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.348521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.350500 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.350479 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xs77v\"" Sep 11 14:15:33.350606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.350486 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 14:15:33.350674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.350659 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 14:15:33.350732 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.350689 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 14:15:33.357845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.357824 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rbkk7"] Sep 11 14:15:33.358758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.358737 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qv6jl"] Sep 11 14:15:33.430194 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.430159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.430369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.430228 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a969f6c-8e60-463f-a6fe-c88428a90c63-tmp-dir\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.430369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.430263 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a969f6c-8e60-463f-a6fe-c88428a90c63-config-volume\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.430369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.430295 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhg8g\" (UniqueName: \"kubernetes.io/projected/6a969f6c-8e60-463f-a6fe-c88428a90c63-kube-api-access-lhg8g\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.531572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a969f6c-8e60-463f-a6fe-c88428a90c63-tmp-dir\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.531745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587pd\" (UniqueName: \"kubernetes.io/projected/59ba9d55-7a02-453f-890b-28a0e451670c-kube-api-access-587pd\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.531745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a969f6c-8e60-463f-a6fe-c88428a90c63-config-volume\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.531745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhg8g\" (UniqueName: \"kubernetes.io/projected/6a969f6c-8e60-463f-a6fe-c88428a90c63-kube-api-access-lhg8g\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.531745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59ba9d55-7a02-453f-890b-28a0e451670c-cert\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.531745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531698 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.532009 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:33.531827 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:33.532009 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:33.531880 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:34.031863488 +0000 UTC m=+36.603520956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:33.532009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.531928 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a969f6c-8e60-463f-a6fe-c88428a90c63-tmp-dir\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.532315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.532297 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a969f6c-8e60-463f-a6fe-c88428a90c63-config-volume\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.543517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.543496 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhg8g\" (UniqueName: \"kubernetes.io/projected/6a969f6c-8e60-463f-a6fe-c88428a90c63-kube-api-access-lhg8g\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:33.632784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.632705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587pd\" (UniqueName: \"kubernetes.io/projected/59ba9d55-7a02-453f-890b-28a0e451670c-kube-api-access-587pd\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.632784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.632767 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59ba9d55-7a02-453f-890b-28a0e451670c-cert\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.635457 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.635435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59ba9d55-7a02-453f-890b-28a0e451670c-cert\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.641805 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.641783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-587pd\" (UniqueName: \"kubernetes.io/projected/59ba9d55-7a02-453f-890b-28a0e451670c-kube-api-access-587pd\") pod \"ingress-canary-qv6jl\" (UID: \"59ba9d55-7a02-453f-890b-28a0e451670c\") " pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:33.664778 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:33.664744 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qv6jl" Sep 11 14:15:34.035832 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.035796 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:34.036462 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:34.035966 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:34.036462 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:34.036038 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:35.036021151 +0000 UTC m=+37.607678602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:34.247030 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.246844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:34.386565 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.386521 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qv6jl"] Sep 11 14:15:34.422003 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:15:34.421976 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ba9d55_7a02_453f_890b_28a0e451670c.slice/crio-39daa1e45a1ef0711714b0aa29050cb8863246cd7a2e966a14dab0fce31aedcd WatchSource:0}: Error finding container 39daa1e45a1ef0711714b0aa29050cb8863246cd7a2e966a14dab0fce31aedcd: Status 404 returned error can't find the container with id 39daa1e45a1ef0711714b0aa29050cb8863246cd7a2e966a14dab0fce31aedcd Sep 11 14:15:34.756815 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.756730 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:34.857159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.857126 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:34.952863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.952832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:15:34.952863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.952863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:15:34.953068 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.952832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.955895 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.956179 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.956200 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.956253 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\"" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.956505 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\"" Sep 11 14:15:34.956831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.956720 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:34.957840 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:34.957816 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:35.040121 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.040091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:35.040580 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:35.040247 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:35.040580 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:35.040316 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:37.04029998 +0000 UTC m=+39.611957431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:35.064367 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.064329 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:35.147089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.147064 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="50a1ca5626c6178842ecd8348ef6606f94266579e7aee76899957664c65c70d4" exitCode=0 Sep 11 14:15:35.147220 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.147146 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"50a1ca5626c6178842ecd8348ef6606f94266579e7aee76899957664c65c70d4"} Sep 11 14:15:35.148344 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.148326 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qv6jl" event={"ID":"59ba9d55-7a02-453f-890b-28a0e451670c","Type":"ContainerStarted","Data":"39daa1e45a1ef0711714b0aa29050cb8863246cd7a2e966a14dab0fce31aedcd"} Sep 11 14:15:35.192738 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.192711 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:35.282795 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:35.282761 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:36.154492 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:36.154458 2571 generic.go:356] "Generic (PLEG): container finished" podID="764a654a-de8b-4e7d-9bd6-029cc4d6f79c" containerID="844ef969643e51dec438b8deb8c8ec433c8b30c01bf4c64e3261ef1e35391a4e" exitCode=0 Sep 11 14:15:36.155033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:36.154522 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerDied","Data":"844ef969643e51dec438b8deb8c8ec433c8b30c01bf4c64e3261ef1e35391a4e"} Sep 11 14:15:36.270446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:36.270418 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:37.054897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.054857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:37.055104 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:37.055057 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:37.055170 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:37.055131 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:41.055115027 +0000 UTC m=+43.626772478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:37.160007 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.159927 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" event={"ID":"764a654a-de8b-4e7d-9bd6-029cc4d6f79c","Type":"ContainerStarted","Data":"02a4f04f6c0482402112783f83e50d2f55810e4b5b008226731ccfd3316a4c4c"} Sep 11 14:15:37.190529 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.190488 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxl7l" podStartSLOduration=8.651800339 podStartE2EDuration="38.190474001s" totalImagesPullingTime="29.538673662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:37.189539863 +0000 UTC m=+39.761197365" watchObservedRunningTime="2026-09-11 14:15:37.190474001 +0000 UTC m=+39.762131473" Sep 11 14:15:37.338825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.338788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:37.442261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.442241 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:37.578757 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:37.578575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:38.163872 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:38.163836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qv6jl" event={"ID":"59ba9d55-7a02-453f-890b-28a0e451670c","Type":"ContainerStarted","Data":"41343826f081a2723487e5d51d65b5cf7c1d6aacc8b10c6b9c02919b0e34ce56"} Sep 11 14:15:38.190412 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:38.190364 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qv6jl" podStartSLOduration=2.20088274 podStartE2EDuration="5.190350072s" totalImagesPullingTime="2.989467332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:38.190201254 +0000 UTC m=+40.761858749" watchObservedRunningTime="2026-09-11 14:15:38.190350072 +0000 UTC m=+40.762007545" Sep 11 14:15:38.418177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:38.418082 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:38.512518 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:38.512485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:38.731422 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:38.731343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:41.080390 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:41.080355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:41.080854 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:41.080504 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:41.080854 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:41.080567 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:49.080552134 +0000 UTC m=+51.652209590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:46.507528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:46.507494 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:46.575705 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:46.575677 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:46.830837 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:46.830801 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:46.900062 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:46.900031 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:49.135032 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:49.134998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:15:49.135471 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:49.135137 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:49.135471 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:15:49.135199 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:05.135181631 +0000 UTC m=+67.706839095 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:15:50.843729 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:50.843681 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:50.912403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:50.912373 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:52.455659 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:52.455627 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:52.533611 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:52.533583 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:59.136820 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:59.136791 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vbvjq" Sep 11 14:15:59.373580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:59.373545 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:59.479898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:59.479804 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:59.573707 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:59.573673 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:59.640483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:15:59.640450 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:03.630375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.630333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:16:03.630798 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.630385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:16:03.632565 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.632543 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:16:03.632646 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.632575 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:16:03.641330 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:03.641301 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 14:16:03.641435 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:03.641375 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs podName:156459d2-1229-42f4-9f9b-09b858d3b4d2 nodeName:}" failed. No retries permitted until 2026-09-11 14:17:07.641353994 +0000 UTC m=+130.213011463 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs") pod "network-metrics-daemon-jnjx5" (UID: "156459d2-1229-42f4-9f9b-09b858d3b4d2") : secret "metrics-daemon-secret" not found Sep 11 14:16:03.645055 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.645030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f74ee825-31fc-4842-8b9f-257b99bab445-original-pull-secret\") pod \"global-pull-secret-syncer-rv7pf\" (UID: \"f74ee825-31fc-4842-8b9f-257b99bab445\") " pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:16:03.731173 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.731133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:16:03.733595 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.733577 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:16:03.743585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.743557 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:16:03.754660 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.754635 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwq4\" (UniqueName: \"kubernetes.io/projected/d54aea61-86a8-436d-9665-0820ffd97b55-kube-api-access-vbwq4\") pod \"network-check-target-drq5n\" (UID: \"d54aea61-86a8-436d-9665-0820ffd97b55\") " pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:16:03.767695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.767672 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\"" Sep 11 14:16:03.775568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.775551 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:16:03.775655 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:03.775553 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv7pf" Sep 11 14:16:04.089839 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:04.089799 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-drq5n"] Sep 11 14:16:04.094039 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:04.093803 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54aea61_86a8_436d_9665_0820ffd97b55.slice/crio-802699d89ed5499e647d931e68b41c91be2b0761279a083ca9ba30b7cc5000c5 WatchSource:0}: Error finding container 802699d89ed5499e647d931e68b41c91be2b0761279a083ca9ba30b7cc5000c5: Status 404 returned error can't find the container with id 802699d89ed5499e647d931e68b41c91be2b0761279a083ca9ba30b7cc5000c5 Sep 11 14:16:04.111952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:04.111905 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv7pf"] Sep 11 14:16:04.115067 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:04.115045 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74ee825_31fc_4842_8b9f_257b99bab445.slice/crio-bbe16a97603ed4de932fab48ad6ef20739f0d7c8758b3204832f5366efd6b06c WatchSource:0}: Error finding container bbe16a97603ed4de932fab48ad6ef20739f0d7c8758b3204832f5366efd6b06c: Status 404 returned error can't find the container with id bbe16a97603ed4de932fab48ad6ef20739f0d7c8758b3204832f5366efd6b06c Sep 11 14:16:04.228061 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:04.227962 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv7pf" event={"ID":"f74ee825-31fc-4842-8b9f-257b99bab445","Type":"ContainerStarted","Data":"bbe16a97603ed4de932fab48ad6ef20739f0d7c8758b3204832f5366efd6b06c"} Sep 11 14:16:04.229119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:04.229096 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-drq5n" event={"ID":"d54aea61-86a8-436d-9665-0820ffd97b55","Type":"ContainerStarted","Data":"802699d89ed5499e647d931e68b41c91be2b0761279a083ca9ba30b7cc5000c5"} Sep 11 14:16:05.139250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:05.139208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:05.139726 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:05.139380 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:16:05.139726 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:05.139450 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls podName:6a969f6c-8e60-463f-a6fe-c88428a90c63 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:37.139432012 +0000 UTC m=+99.711089476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls") pod "dns-default-rbkk7" (UID: "6a969f6c-8e60-463f-a6fe-c88428a90c63") : secret "dns-default-metrics-tls" not found Sep 11 14:16:07.240781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:07.240694 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-drq5n" event={"ID":"d54aea61-86a8-436d-9665-0820ffd97b55","Type":"ContainerStarted","Data":"c918068bf04214c3f794e966eb7a0fb7b836ea3f1f0d83f29b7eaed3c2e9cf66"} Sep 11 14:16:07.241240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:07.240861 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:16:07.265598 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:07.263136 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-drq5n" podStartSLOduration=65.326080832 podStartE2EDuration="1m8.263117364s" totalImagesPullingTime="2.937036532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:07.261812309 +0000 UTC m=+69.833469793" watchObservedRunningTime="2026-09-11 14:16:07.263117364 +0000 UTC m=+69.834774840" Sep 11 14:16:09.300200 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.300167 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v"] Sep 11 14:16:09.302917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.302902 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" Sep 11 14:16:09.305585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.305558 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:09.305701 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.305561 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:09.306016 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.305996 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pzfzn\"" Sep 11 14:16:09.311053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.311033 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-g4f5j"] Sep 11 14:16:09.313571 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.313555 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.316230 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.316212 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 14:16:09.316380 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.316358 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 14:16:09.317001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.316663 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v"] Sep 11 14:16:09.317001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.316816 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 14:16:09.317211 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.317069 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g6zrn\"" Sep 11 14:16:09.317211 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.317116 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 14:16:09.321275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.321253 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 14:16:09.336001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.335977 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-g4f5j"] Sep 11 14:16:09.372013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.371981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr775\" (UniqueName: \"kubernetes.io/projected/b29c3578-ebff-44bf-b071-4819ce9f1a2e-kube-api-access-rr775\") pod \"volume-data-source-validator-86695b7cc7-wqd5v\" (UID: \"b29c3578-ebff-44bf-b071-4819ce9f1a2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" Sep 11 14:16:09.372013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-service-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.372203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372030 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h96\" (UniqueName: \"kubernetes.io/projected/f1380412-5d49-4c33-b973-cf9cabe5f316-kube-api-access-w4h96\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.372203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372052 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-snapshots\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.372203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372122 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-tmp\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.372203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-trusted-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.372462 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.372206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1380412-5d49-4c33-b973-cf9cabe5f316-serving-cert\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.472783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-snapshots\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.472783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-tmp\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.472983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472864 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-trusted-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.472983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472901 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1380412-5d49-4c33-b973-cf9cabe5f316-serving-cert\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.472983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr775\" (UniqueName: \"kubernetes.io/projected/b29c3578-ebff-44bf-b071-4819ce9f1a2e-kube-api-access-rr775\") pod \"volume-data-source-validator-86695b7cc7-wqd5v\" (UID: \"b29c3578-ebff-44bf-b071-4819ce9f1a2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" Sep 11 14:16:09.472983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.472982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-service-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.473176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.473001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h96\" (UniqueName: \"kubernetes.io/projected/f1380412-5d49-4c33-b973-cf9cabe5f316-kube-api-access-w4h96\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.473176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.473104 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-tmp\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.473466 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.473442 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f1380412-5d49-4c33-b973-cf9cabe5f316-snapshots\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.473528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.473470 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-service-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.473759 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.473741 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1380412-5d49-4c33-b973-cf9cabe5f316-trusted-ca-bundle\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.475203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.475181 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1380412-5d49-4c33-b973-cf9cabe5f316-serving-cert\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.481965 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.481924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr775\" (UniqueName: \"kubernetes.io/projected/b29c3578-ebff-44bf-b071-4819ce9f1a2e-kube-api-access-rr775\") pod \"volume-data-source-validator-86695b7cc7-wqd5v\" (UID: \"b29c3578-ebff-44bf-b071-4819ce9f1a2e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" Sep 11 14:16:09.482120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.482103 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h96\" (UniqueName: \"kubernetes.io/projected/f1380412-5d49-4c33-b973-cf9cabe5f316-kube-api-access-w4h96\") pod \"insights-operator-5844b856db-g4f5j\" (UID: \"f1380412-5d49-4c33-b973-cf9cabe5f316\") " pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.533019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.532982 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:09.613074 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.613046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" Sep 11 14:16:09.624014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.623992 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-g4f5j" Sep 11 14:16:09.645524 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.645492 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:09.884410 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.884378 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v"] Sep 11 14:16:09.889388 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:09.889360 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29c3578_ebff_44bf_b071_4819ce9f1a2e.slice/crio-1773fada56b1faf53e14321f84d5a1f3f8ea2040ef93512105cb8398787ac37d WatchSource:0}: Error finding container 1773fada56b1faf53e14321f84d5a1f3f8ea2040ef93512105cb8398787ac37d: Status 404 returned error can't find the container with id 1773fada56b1faf53e14321f84d5a1f3f8ea2040ef93512105cb8398787ac37d Sep 11 14:16:09.906684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:09.906662 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-g4f5j"] Sep 11 14:16:09.909490 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:09.909468 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1380412_5d49_4c33_b973_cf9cabe5f316.slice/crio-3f36c238f6d33cc274f869e360281fce92f09192c955f3db077b274b8afc4262 WatchSource:0}: Error finding container 3f36c238f6d33cc274f869e360281fce92f09192c955f3db077b274b8afc4262: Status 404 returned error can't find the container with id 3f36c238f6d33cc274f869e360281fce92f09192c955f3db077b274b8afc4262 Sep 11 14:16:10.249614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:10.249533 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-g4f5j" event={"ID":"f1380412-5d49-4c33-b973-cf9cabe5f316","Type":"ContainerStarted","Data":"3f36c238f6d33cc274f869e360281fce92f09192c955f3db077b274b8afc4262"} Sep 11 14:16:10.250688 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:10.250661 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" event={"ID":"b29c3578-ebff-44bf-b071-4819ce9f1a2e","Type":"ContainerStarted","Data":"1773fada56b1faf53e14321f84d5a1f3f8ea2040ef93512105cb8398787ac37d"} Sep 11 14:16:10.252051 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:10.252030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv7pf" event={"ID":"f74ee825-31fc-4842-8b9f-257b99bab445","Type":"ContainerStarted","Data":"cecc5d8510b4cc8ea2413ad7a3012001736c1340b7e86cf4ddd342c10e7063f6"} Sep 11 14:16:10.270096 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:10.270055 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rv7pf" podStartSLOduration=66.218151377 podStartE2EDuration="1m11.270039893s" totalImagesPullingTime="5.051888516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:10.269465579 +0000 UTC m=+72.841123053" watchObservedRunningTime="2026-09-11 14:16:10.270039893 +0000 UTC m=+72.841697423" Sep 11 14:16:11.209106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.209034 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bc49w"] Sep 11 14:16:11.213372 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.213351 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.215678 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.215659 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4v2cp\"" Sep 11 14:16:11.215785 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.215659 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 14:16:11.216004 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.215986 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 14:16:11.216102 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.216022 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:11.216102 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.216039 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:11.222390 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.222367 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 14:16:11.224888 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.224866 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bc49w"] Sep 11 14:16:11.256706 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.256675 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" event={"ID":"b29c3578-ebff-44bf-b071-4819ce9f1a2e","Type":"ContainerStarted","Data":"f3bad16b9434941699348b1522a2a178c0d8a5d7583dcb82f7f94bb3cce6dedc"} Sep 11 14:16:11.286042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.286013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-serving-cert\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.286185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.286096 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-config\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.286266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.286216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-trusted-ca\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.286266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.286245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59ws\" (UniqueName: \"kubernetes.io/projected/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-kube-api-access-d59ws\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.387413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.387374 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-serving-cert\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.387413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.387413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-config\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.387628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.387435 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-trusted-ca\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.387628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.387456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59ws\" (UniqueName: \"kubernetes.io/projected/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-kube-api-access-d59ws\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.388203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.388172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-config\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.388404 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.388386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-trusted-ca\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.390201 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.390180 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-serving-cert\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.397798 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.397772 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59ws\" (UniqueName: \"kubernetes.io/projected/fe0cc6a4-21f9-401c-8b7a-c76c4fb890db-kube-api-access-d59ws\") pod \"console-operator-696884fb9c-bc49w\" (UID: \"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db\") " pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.400142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.400044 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:11.479616 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.479463 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:11.524393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.524361 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:11.819211 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.819123 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-wqd5v" podStartSLOduration=1.821642458 podStartE2EDuration="2.819106282s" totalImagesPullingTime="997.463824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:11.285743447 +0000 UTC m=+73.857400946" watchObservedRunningTime="2026-09-11 14:16:11.819106282 +0000 UTC m=+74.390763800" Sep 11 14:16:11.819886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:11.819863 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bc49w"] Sep 11 14:16:11.823284 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:11.823261 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0cc6a4_21f9_401c_8b7a_c76c4fb890db.slice/crio-046c8e542dfcdd88306598b0ec239ffadc2c128dbed9ae282225239d760f0573 WatchSource:0}: Error finding container 046c8e542dfcdd88306598b0ec239ffadc2c128dbed9ae282225239d760f0573: Status 404 returned error can't find the container with id 046c8e542dfcdd88306598b0ec239ffadc2c128dbed9ae282225239d760f0573 Sep 11 14:16:12.248568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.248480 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pkghv"] Sep 11 14:16:12.249453 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.249426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.251852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.251834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.254651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.254620 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 14:16:12.254651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.254629 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 14:16:12.254786 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.254661 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rj6zn\"" Sep 11 14:16:12.261923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.261888 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" event={"ID":"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db","Type":"ContainerStarted","Data":"046c8e542dfcdd88306598b0ec239ffadc2c128dbed9ae282225239d760f0573"} Sep 11 14:16:12.263595 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.263570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-g4f5j" event={"ID":"f1380412-5d49-4c33-b973-cf9cabe5f316","Type":"ContainerStarted","Data":"29cd1312b393c40ccb3738a01e1178aba6c7b580e78932ca7b3c9714a9285d4f"} Sep 11 14:16:12.271095 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.271071 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkghv"] Sep 11 14:16:12.295241 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.295209 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/982c0ac4-150b-4d33-a391-9f70c3abaedb-data-volume\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.295407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.295248 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.295407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.295320 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.295578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.295554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/982c0ac4-150b-4d33-a391-9f70c3abaedb-crio-socket\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.295657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.295592 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fvb\" (UniqueName: \"kubernetes.io/projected/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-api-access-p9fvb\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.315287 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.315235 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-g4f5j" podStartSLOduration=1.677406973 podStartE2EDuration="3.315217725s" totalImagesPullingTime="1.637810752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:12.314723587 +0000 UTC m=+74.886381082" watchObservedRunningTime="2026-09-11 14:16:12.315217725 +0000 UTC m=+74.886875247" Sep 11 14:16:12.374656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.374620 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.396743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.396711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/982c0ac4-150b-4d33-a391-9f70c3abaedb-data-volume\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.396898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.396748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.396898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.396788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.396898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.396849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/982c0ac4-150b-4d33-a391-9f70c3abaedb-crio-socket\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.396898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.396878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fvb\" (UniqueName: \"kubernetes.io/projected/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-api-access-p9fvb\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.397138 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.397017 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:12.397138 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.397107 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.897086404 +0000 UTC m=+75.468743861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : configmap "kube-rbac-proxy" not found Sep 11 14:16:12.397138 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.397107 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/982c0ac4-150b-4d33-a391-9f70c3abaedb-data-volume\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.397303 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.397159 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/982c0ac4-150b-4d33-a391-9f70c3abaedb-crio-socket\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.397362 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.397307 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.397412 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.397378 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.897363222 +0000 UTC m=+75.469020685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.423209 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.423176 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fvb\" (UniqueName: \"kubernetes.io/projected/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-api-access-p9fvb\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.808042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.808003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.880061 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.880014 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.902078 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.902046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.902267 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.902178 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:12.902267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.902213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:12.902267 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.902244 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.902225061 +0000 UTC m=+76.473882520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : configmap "kube-rbac-proxy" not found Sep 11 14:16:12.902397 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.902289 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.902397 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:12.902332 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.902320964 +0000 UTC m=+76.473978429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.958621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:12.958555 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.908244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:13.908208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:13.908244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:13.908249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:13.908746 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:13.908333 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:13.908746 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:13.908382 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.908369442 +0000 UTC m=+78.480026892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : configmap "kube-rbac-proxy" not found Sep 11 14:16:13.908746 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:13.908379 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.908746 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:13.908451 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.908433467 +0000 UTC m=+78.480090919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:14.190686 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.190617 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5dl9d_be04675b-fc49-48cc-9121-3a49b2ecb6fb/dns-node-resolver/0.log" Sep 11 14:16:14.273107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.273081 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/0.log" Sep 11 14:16:14.273276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.273116 2571 generic.go:356] "Generic (PLEG): container finished" podID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" containerID="785bbbf6704ab457e7948521bd4d8dbe10dfbf679fe09a7e32cc9957a2ca2c05" exitCode=255 Sep 11 14:16:14.273276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.273156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" event={"ID":"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db","Type":"ContainerDied","Data":"785bbbf6704ab457e7948521bd4d8dbe10dfbf679fe09a7e32cc9957a2ca2c05"} Sep 11 14:16:14.273414 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.273398 2571 scope.go:117] "RemoveContainer" containerID="785bbbf6704ab457e7948521bd4d8dbe10dfbf679fe09a7e32cc9957a2ca2c05" Sep 11 14:16:14.991130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:14.991100 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n228s_b30ec984-eb09-4f9e-aa21-d12eaa0e73ad/node-ca/0.log" Sep 11 14:16:15.278388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.278363 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/1.log" Sep 11 14:16:15.278794 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.278775 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/0.log" Sep 11 14:16:15.278884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.278814 2571 generic.go:356] "Generic (PLEG): container finished" podID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" exitCode=255 Sep 11 14:16:15.278884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.278863 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" event={"ID":"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db","Type":"ContainerDied","Data":"b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825"} Sep 11 14:16:15.278997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.278965 2571 scope.go:117] "RemoveContainer" containerID="785bbbf6704ab457e7948521bd4d8dbe10dfbf679fe09a7e32cc9957a2ca2c05" Sep 11 14:16:15.279184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.279168 2571 scope.go:117] "RemoveContainer" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" Sep 11 14:16:15.279347 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:15.279327 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bc49w_openshift-console-operator(fe0cc6a4-21f9-401c-8b7a-c76c4fb890db)\"" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podUID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" Sep 11 14:16:15.788471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.788441 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qv6jl_59ba9d55-7a02-453f-890b-28a0e451670c/serve-healthcheck-canary/0.log" Sep 11 14:16:15.924584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.924537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:15.924584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:15.924591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:15.924836 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:15.924706 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.924836 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:15.924758 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:15.924836 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:15.924783 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.924763704 +0000 UTC m=+82.496421169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.924836 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:15.924804 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.92479187 +0000 UTC m=+82.496449333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : configmap "kube-rbac-proxy" not found Sep 11 14:16:16.283637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:16.283611 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/1.log" Sep 11 14:16:16.284034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:16.283922 2571 scope.go:117] "RemoveContainer" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" Sep 11 14:16:16.284105 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:16.284085 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bc49w_openshift-console-operator(fe0cc6a4-21f9-401c-8b7a-c76c4fb890db)\"" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podUID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" Sep 11 14:16:17.201222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.201188 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m"] Sep 11 14:16:17.205166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.205150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" Sep 11 14:16:17.211752 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.211736 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nw2wp\"" Sep 11 14:16:17.229173 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.229149 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m"] Sep 11 14:16:17.233918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.233894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszkt\" (UniqueName: \"kubernetes.io/projected/00cc4f2b-b43c-4ad9-bf4c-99c06c874007-kube-api-access-mszkt\") pod \"network-check-source-6f8b768455-xqj8m\" (UID: \"00cc4f2b-b43c-4ad9-bf4c-99c06c874007\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" Sep 11 14:16:17.304697 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.304670 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5"] Sep 11 14:16:17.307603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.307588 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.310977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.310956 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 14:16:17.310977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.310952 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:17.311131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.311098 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 14:16:17.311202 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.311186 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:17.311245 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.311210 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kmqsx\"" Sep 11 14:16:17.331915 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.331892 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5"] Sep 11 14:16:17.335306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.335284 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszkt\" (UniqueName: \"kubernetes.io/projected/00cc4f2b-b43c-4ad9-bf4c-99c06c874007-kube-api-access-mszkt\") pod \"network-check-source-6f8b768455-xqj8m\" (UID: \"00cc4f2b-b43c-4ad9-bf4c-99c06c874007\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" Sep 11 14:16:17.335400 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.335339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.335459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.335396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdlm\" (UniqueName: \"kubernetes.io/projected/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-kube-api-access-4xdlm\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.335459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.335427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-config\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.349958 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.349926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszkt\" (UniqueName: \"kubernetes.io/projected/00cc4f2b-b43c-4ad9-bf4c-99c06c874007-kube-api-access-mszkt\") pod \"network-check-source-6f8b768455-xqj8m\" (UID: \"00cc4f2b-b43c-4ad9-bf4c-99c06c874007\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" Sep 11 14:16:17.436473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.436443 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdlm\" (UniqueName: \"kubernetes.io/projected/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-kube-api-access-4xdlm\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.436473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.436478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-config\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.436670 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.436543 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.437116 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.437087 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-config\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.438627 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.438609 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-serving-cert\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.450194 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.450172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdlm\" (UniqueName: \"kubernetes.io/projected/c68b6b56-9dd9-4050-963f-39ce4edc9cbf-kube-api-access-4xdlm\") pod \"service-ca-operator-cf7c776b7-tdrt5\" (UID: \"c68b6b56-9dd9-4050-963f-39ce4edc9cbf\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.513034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.512964 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" Sep 11 14:16:17.615394 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.615363 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" Sep 11 14:16:17.792560 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.792533 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m"] Sep 11 14:16:17.795632 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:17.795600 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cc4f2b_b43c_4ad9_bf4c_99c06c874007.slice/crio-942eef0557e55ce728311162d3d1ee80573b8ffeab4fe882f9b43a6d3d16fc4a WatchSource:0}: Error finding container 942eef0557e55ce728311162d3d1ee80573b8ffeab4fe882f9b43a6d3d16fc4a: Status 404 returned error can't find the container with id 942eef0557e55ce728311162d3d1ee80573b8ffeab4fe882f9b43a6d3d16fc4a Sep 11 14:16:17.895387 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:17.895352 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5"] Sep 11 14:16:17.898997 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:17.898959 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68b6b56_9dd9_4050_963f_39ce4edc9cbf.slice/crio-6514d9584073ae3f546a14f5db654b30439e13db0dd6f1bb6309d054e65a8be9 WatchSource:0}: Error finding container 6514d9584073ae3f546a14f5db654b30439e13db0dd6f1bb6309d054e65a8be9: Status 404 returned error can't find the container with id 6514d9584073ae3f546a14f5db654b30439e13db0dd6f1bb6309d054e65a8be9 Sep 11 14:16:18.053898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.053810 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:18.116084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.116058 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk"] Sep 11 14:16:18.119196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.119181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" Sep 11 14:16:18.121280 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.121261 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:18.121369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.121261 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hhcns\"" Sep 11 14:16:18.121369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.121300 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 11 14:16:18.144606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.144583 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk"] Sep 11 14:16:18.154486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.154453 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:18.243568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.243526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7ww\" (UniqueName: \"kubernetes.io/projected/7c2fa79c-afc9-457a-9789-63db70c0a265-kube-api-access-4k7ww\") pod \"migrator-8b4d697c5-t8jfk\" (UID: \"7c2fa79c-afc9-457a-9789-63db70c0a265\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" Sep 11 14:16:18.249539 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.249509 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:18.294275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.294238 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" event={"ID":"00cc4f2b-b43c-4ad9-bf4c-99c06c874007","Type":"ContainerStarted","Data":"248a3bd7c937a0e2b79794e52bd68de18d8f448035e066d4edf9a7da758325e0"} Sep 11 14:16:18.294421 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.294294 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" event={"ID":"00cc4f2b-b43c-4ad9-bf4c-99c06c874007","Type":"ContainerStarted","Data":"942eef0557e55ce728311162d3d1ee80573b8ffeab4fe882f9b43a6d3d16fc4a"} Sep 11 14:16:18.296034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.296008 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" event={"ID":"c68b6b56-9dd9-4050-963f-39ce4edc9cbf","Type":"ContainerStarted","Data":"6514d9584073ae3f546a14f5db654b30439e13db0dd6f1bb6309d054e65a8be9"} Sep 11 14:16:18.316831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.316750 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-xqj8m" podStartSLOduration=1.31673493 podStartE2EDuration="1.31673493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:18.316026433 +0000 UTC m=+80.887683922" watchObservedRunningTime="2026-09-11 14:16:18.31673493 +0000 UTC m=+80.888392403" Sep 11 14:16:18.344619 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.344591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7ww\" (UniqueName: \"kubernetes.io/projected/7c2fa79c-afc9-457a-9789-63db70c0a265-kube-api-access-4k7ww\") pod \"migrator-8b4d697c5-t8jfk\" (UID: \"7c2fa79c-afc9-457a-9789-63db70c0a265\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" Sep 11 14:16:18.361071 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.361025 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7ww\" (UniqueName: \"kubernetes.io/projected/7c2fa79c-afc9-457a-9789-63db70c0a265-kube-api-access-4k7ww\") pod \"migrator-8b4d697c5-t8jfk\" (UID: \"7c2fa79c-afc9-457a-9789-63db70c0a265\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" Sep 11 14:16:18.428037 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.428009 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" Sep 11 14:16:18.713663 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:18.713507 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk"] Sep 11 14:16:18.716620 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:18.716586 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2fa79c_afc9_457a_9789_63db70c0a265.slice/crio-27fc74e704c39491f099a1dd245d280c4863ffe56c110e98d6fb9cf623a82418 WatchSource:0}: Error finding container 27fc74e704c39491f099a1dd245d280c4863ffe56c110e98d6fb9cf623a82418: Status 404 returned error can't find the container with id 27fc74e704c39491f099a1dd245d280c4863ffe56c110e98d6fb9cf623a82418 Sep 11 14:16:19.301547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:19.301463 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" event={"ID":"7c2fa79c-afc9-457a-9789-63db70c0a265","Type":"ContainerStarted","Data":"27fc74e704c39491f099a1dd245d280c4863ffe56c110e98d6fb9cf623a82418"} Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:19.958169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:19.958234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:19.958356 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:19.958403 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.95838636 +0000 UTC m=+90.530043817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : configmap "kube-rbac-proxy" not found Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:19.958477 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:19.960351 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:19.958508 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.95849856 +0000 UTC m=+90.530156025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:20.306110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.306074 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" event={"ID":"7c2fa79c-afc9-457a-9789-63db70c0a265","Type":"ContainerStarted","Data":"361e4fac7dde349f98855290dba7e7578040c826c2e9df98d37e5f03e133ade4"} Sep 11 14:16:20.306249 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.306174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" event={"ID":"7c2fa79c-afc9-457a-9789-63db70c0a265","Type":"ContainerStarted","Data":"488cef26e1510fec720d0d2c418214df91086a1e78f93582cd574eb69dd82742"} Sep 11 14:16:20.307609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.307587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" event={"ID":"c68b6b56-9dd9-4050-963f-39ce4edc9cbf","Type":"ContainerStarted","Data":"e509339d0e09f73294171fdabedceace7c0d59d0e40f233fb7babd2269a94d81"} Sep 11 14:16:20.324084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.324051 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-t8jfk" podStartSLOduration=1.150901515 podStartE2EDuration="2.324039536s" totalImagesPullingTime="1.173138021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:20.323358013 +0000 UTC m=+82.895015498" watchObservedRunningTime="2026-09-11 14:16:20.324039536 +0000 UTC m=+82.895697049" Sep 11 14:16:20.341473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.341438 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" podStartSLOduration=1.349334627 podStartE2EDuration="3.341425206s" totalImagesPullingTime="1.992090579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:20.34065422 +0000 UTC m=+82.912311693" watchObservedRunningTime="2026-09-11 14:16:20.341425206 +0000 UTC m=+82.913082679" Sep 11 14:16:20.427758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.427728 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:20.497847 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:20.497821 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.371984 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.371956 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.461138 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.461101 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.525091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.525068 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:21.525214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.525108 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:21.525442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.525426 2571 scope.go:117] "RemoveContainer" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" Sep 11 14:16:21.525581 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:21.525563 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bc49w_openshift-console-operator(fe0cc6a4-21f9-401c-8b7a-c76c4fb890db)\"" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podUID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" Sep 11 14:16:21.822910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.822874 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.898732 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:21.898696 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.087147 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:22.087063 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.179881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:22.179850 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.721742 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:23.721711 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.792951 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:23.792902 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:25.396313 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:25.396280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:26.506833 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:26.506802 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.173515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.173486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.482884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.482797 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.570186 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.570148 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.666278 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.666248 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.738569 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.738487 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.943399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:27.943367 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:28.018856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:28.018772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:28.018856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:28.018818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:28.019080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:28.018922 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:28.019080 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:28.019019 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls podName:982c0ac4-150b-4d33-a391-9f70c3abaedb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:44.018997238 +0000 UTC m=+106.590654695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkghv" (UID: "982c0ac4-150b-4d33-a391-9f70c3abaedb") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:28.019489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:28.019467 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/982c0ac4-150b-4d33-a391-9f70c3abaedb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:28.080215 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:28.080175 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:28.280577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:28.280536 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.293031 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:29.293001 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.470654 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:29.470620 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.669376 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:29.669275 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.901041 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:29.901013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:30.072114 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:30.072083 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:30.854768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:30.854728 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:30.924243 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:30.924212 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.076201 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.076169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.463875 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.463843 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.676163 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.676125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.756623 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.756542 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.837685 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.837649 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.906342 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:31.906312 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:32.272515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:32.272475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:32.686887 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:32.686858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:33.953176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:33.953147 2571 scope.go:117] "RemoveContainer" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" Sep 11 14:16:34.163988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.163955 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.262514 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.262436 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.351917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.351893 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:16:34.352336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.352321 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/1.log" Sep 11 14:16:34.352382 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.352354 2571 generic.go:356] "Generic (PLEG): container finished" podID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" containerID="2c81e129c66892622086447e52e02b46474d94be8ea1fed09af1801feffa1949" exitCode=255 Sep 11 14:16:34.352436 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.352422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" event={"ID":"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db","Type":"ContainerDied","Data":"2c81e129c66892622086447e52e02b46474d94be8ea1fed09af1801feffa1949"} Sep 11 14:16:34.352481 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.352469 2571 scope.go:117] "RemoveContainer" containerID="b81388af733bb9c13f4f5aaca4b032535e336da30994f38870eb30116ceed825" Sep 11 14:16:34.352785 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.352751 2571 scope.go:117] "RemoveContainer" containerID="2c81e129c66892622086447e52e02b46474d94be8ea1fed09af1801feffa1949" Sep 11 14:16:34.352947 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:34.352917 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bc49w_openshift-console-operator(fe0cc6a4-21f9-401c-8b7a-c76c4fb890db)\"" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podUID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" Sep 11 14:16:34.673883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:34.673847 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:35.073066 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:35.073033 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:35.264391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:35.264356 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:35.356853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:35.356780 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:16:35.666262 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:35.666136 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:36.268576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:36.268543 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:36.678606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:36.678569 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:36.870595 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:36.870562 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:37.171558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.171530 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:37.189625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.189601 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:37.191717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.191694 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a969f6c-8e60-463f-a6fe-c88428a90c63-metrics-tls\") pod \"dns-default-rbkk7\" (UID: \"6a969f6c-8e60-463f-a6fe-c88428a90c63\") " pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:37.259270 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.259241 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\"" Sep 11 14:16:37.267279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.267258 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:37.474612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.474540 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:37.570184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.570156 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rbkk7"] Sep 11 14:16:37.572794 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:37.572761 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a969f6c_8e60_463f_a6fe_c88428a90c63.slice/crio-e703ee4b896cd125120ec5cb40773a80b8697a14fca9ca0f53044ed7c50b2a04 WatchSource:0}: Error finding container e703ee4b896cd125120ec5cb40773a80b8697a14fca9ca0f53044ed7c50b2a04: Status 404 returned error can't find the container with id e703ee4b896cd125120ec5cb40773a80b8697a14fca9ca0f53044ed7c50b2a04 Sep 11 14:16:37.671483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.671443 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:37.868218 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:37.868182 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:38.246529 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:38.246451 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-drq5n" Sep 11 14:16:38.367868 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:38.367831 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbkk7" event={"ID":"6a969f6c-8e60-463f-a6fe-c88428a90c63","Type":"ContainerStarted","Data":"e703ee4b896cd125120ec5cb40773a80b8697a14fca9ca0f53044ed7c50b2a04"} Sep 11 14:16:39.373224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:39.373140 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbkk7" event={"ID":"6a969f6c-8e60-463f-a6fe-c88428a90c63","Type":"ContainerStarted","Data":"089002eb1710bb83c9bdc5ef854e1071d39f7e70804a27fa4fbf1ee7a8c9ad26"} Sep 11 14:16:39.373224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:39.373188 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbkk7" event={"ID":"6a969f6c-8e60-463f-a6fe-c88428a90c63","Type":"ContainerStarted","Data":"af333c84fdcbbf81e05877f80472fdfdc9ab4925c557d2f5bdd67ad928577d01"} Sep 11 14:16:39.373629 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:39.373239 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:39.413295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:39.413252 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rbkk7" podStartSLOduration=64.894916171 podStartE2EDuration="1m6.413237428s" totalImagesPullingTime="1.518321257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:39.412621497 +0000 UTC m=+101.984278972" watchObservedRunningTime="2026-09-11 14:16:39.413237428 +0000 UTC m=+101.984894901" Sep 11 14:16:41.524681 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:41.524637 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:41.525131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:41.524725 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:41.525131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:41.525029 2571 scope.go:117] "RemoveContainer" containerID="2c81e129c66892622086447e52e02b46474d94be8ea1fed09af1801feffa1949" Sep 11 14:16:41.525234 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:41.525163 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bc49w_openshift-console-operator(fe0cc6a4-21f9-401c-8b7a-c76c4fb890db)\"" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podUID="fe0cc6a4-21f9-401c-8b7a-c76c4fb890db" Sep 11 14:16:44.033508 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.033475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:44.036613 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.036595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:44.038863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.038838 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/982c0ac4-150b-4d33-a391-9f70c3abaedb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkghv\" (UID: \"982c0ac4-150b-4d33-a391-9f70c3abaedb\") " pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:44.062948 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.062911 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkghv" Sep 11 14:16:44.143550 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.143524 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:44.279655 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.279623 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:44.401329 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.401262 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkghv"] Sep 11 14:16:44.404384 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:44.404359 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982c0ac4_150b_4d33_a391_9f70c3abaedb.slice/crio-f3259cd1efa983fe11e1f5a562d8fcae22ab438d26878644b91bb008d0217549 WatchSource:0}: Error finding container f3259cd1efa983fe11e1f5a562d8fcae22ab438d26878644b91bb008d0217549: Status 404 returned error can't find the container with id f3259cd1efa983fe11e1f5a562d8fcae22ab438d26878644b91bb008d0217549 Sep 11 14:16:44.423661 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:44.423627 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:45.395269 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:45.395231 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkghv" event={"ID":"982c0ac4-150b-4d33-a391-9f70c3abaedb","Type":"ContainerStarted","Data":"d4d5172257dbf8325eb39d883e7fe37467a34b5fed44bdfd11e1a728ccf5fdbe"} Sep 11 14:16:45.395623 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:45.395287 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkghv" event={"ID":"982c0ac4-150b-4d33-a391-9f70c3abaedb","Type":"ContainerStarted","Data":"f3259cd1efa983fe11e1f5a562d8fcae22ab438d26878644b91bb008d0217549"} Sep 11 14:16:46.404853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:46.404813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkghv" event={"ID":"982c0ac4-150b-4d33-a391-9f70c3abaedb","Type":"ContainerStarted","Data":"97f9c464e535f470e871c6011eccd8b5fbde9b3700e830236f205c3c1c1e932c"} Sep 11 14:16:46.852428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:46.852379 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:46.956187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:46.956140 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:47.223919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.223891 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84db9b5846-s5dwb"] Sep 11 14:16:47.226780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.226766 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.255027 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.254990 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 14:16:47.255121 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.255065 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 14:16:47.255198 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.255137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-92zwk\"" Sep 11 14:16:47.255255 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.255243 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 14:16:47.267323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.267296 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 14:16:47.316206 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.316182 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84db9b5846-s5dwb"] Sep 11 14:16:47.361668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361612 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw"] Sep 11 14:16:47.361836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.361878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmdk\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.361915 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361875 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.361915 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.361915 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.361909 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.362050 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.362028 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.362108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.362092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.362157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.362127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.364551 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.364537 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.369230 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.369204 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:16:47.369230 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.369206 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Sep 11 14:16:47.369361 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.369246 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:16:47.377632 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.377608 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"work-manager-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" Sep 11 14:16:47.392832 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.392804 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" podUID="8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.409509 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.409484 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkghv" event={"ID":"982c0ac4-150b-4d33-a391-9f70c3abaedb","Type":"ContainerStarted","Data":"48ec0d8b2cc00e9afe022fa706fb8a762b8f6eadbb190ad05d2e06422290f245"} Sep 11 14:16:47.446889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.446868 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw"] Sep 11 14:16:47.462694 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.462762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-tmp\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.462762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.462762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.462895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.462895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462837 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.462895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462860 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmdk\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.462895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462884 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.463136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.463136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.462951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.463136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.463060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.463645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.463622 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.463732 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.463628 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.463878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.463860 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.465219 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.465199 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.465649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.465629 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.465761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.465741 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.501122 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.501101 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84db9b5846-s5dwb"] Sep 11 14:16:47.501277 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:47.501261 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token kube-api-access-5lmdk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" podUID="20116020-6f5c-43c8-8fb6-d4f5c3833d5c" Sep 11 14:16:47.515739 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.515709 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.529288 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.529265 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmdk\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk\") pod \"image-registry-84db9b5846-s5dwb\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:47.564333 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.564308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-tmp\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.564433 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.564367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.564433 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.564418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.564776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.564753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-tmp\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:47.570235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.570216 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f7f6bb8f9-khnkp"] Sep 11 14:16:47.574393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.574379 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.613538 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.613472 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pkghv" podStartSLOduration=33.156650826 podStartE2EDuration="35.61345954s" totalImagesPullingTime="2.456808714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:47.601145109 +0000 UTC m=+110.172802583" watchObservedRunningTime="2026-09-11 14:16:47.61345954 +0000 UTC m=+110.185117013" Sep 11 14:16:47.614224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.614208 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f7f6bb8f9-khnkp"] Sep 11 14:16:47.664991 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.664970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-trusted-ca\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-certificates\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665073 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-bound-sa-token\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-tls\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb00318b-3c1b-49b2-8bd6-5f7403813874-ca-trust-extracted\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-image-registry-private-configuration\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwntn\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-kube-api-access-fwntn\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-installation-pull-secrets\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.665994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.665973 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:47.766275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-certificates\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-bound-sa-token\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766299 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-tls\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb00318b-3c1b-49b2-8bd6-5f7403813874-ca-trust-extracted\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766347 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-image-registry-private-configuration\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwntn\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-kube-api-access-fwntn\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766650 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-installation-pull-secrets\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766650 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-trusted-ca\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.766750 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.766731 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb00318b-3c1b-49b2-8bd6-5f7403813874-ca-trust-extracted\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.767142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.767115 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-certificates\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.767403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.767381 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb00318b-3c1b-49b2-8bd6-5f7403813874-trusted-ca\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.768911 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.768885 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-image-registry-private-configuration\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.769122 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.769098 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-registry-tls\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.769182 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.769166 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb00318b-3c1b-49b2-8bd6-5f7403813874-installation-pull-secrets\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.786395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.786376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-bound-sa-token\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.791862 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.791845 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwntn\" (UniqueName: \"kubernetes.io/projected/bb00318b-3c1b-49b2-8bd6-5f7403813874-kube-api-access-fwntn\") pod \"image-registry-6f7f6bb8f9-khnkp\" (UID: \"bb00318b-3c1b-49b2-8bd6-5f7403813874\") " pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:47.823238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.823216 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:47.882796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:47.882747 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:48.185859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.185837 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f7f6bb8f9-khnkp"] Sep 11 14:16:48.187700 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:48.187669 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb00318b_3c1b_49b2_8bd6_5f7403813874.slice/crio-bd0aca34049d147641bd9735379b8e5c75990e4b5ac478a8e1f158299f7e165c WatchSource:0}: Error finding container bd0aca34049d147641bd9735379b8e5c75990e4b5ac478a8e1f158299f7e165c: Status 404 returned error can't find the container with id bd0aca34049d147641bd9735379b8e5c75990e4b5ac478a8e1f158299f7e165c Sep 11 14:16:48.326329 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.326295 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:16:48.414896 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.414851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:48.415318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.414856 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" event={"ID":"bb00318b-3c1b-49b2-8bd6-5f7403813874","Type":"ContainerStarted","Data":"a20ec33b78d86e6d4c9292d1fd154884b55da8b8f5a8884d944a4d473f63e345"} Sep 11 14:16:48.415318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.414987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" event={"ID":"bb00318b-3c1b-49b2-8bd6-5f7403813874","Type":"ContainerStarted","Data":"bd0aca34049d147641bd9735379b8e5c75990e4b5ac478a8e1f158299f7e165c"} Sep 11 14:16:48.415318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.415195 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:16:48.421038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.421020 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:48.565319 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:48.565286 2571 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/work-manager-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:48.565495 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:48.565376 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config podName:8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:49.065355272 +0000 UTC m=+111.637012737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "klusterlet-config" (UniqueName: "kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config") pod "klusterlet-addon-workmgr-5455fc8d7b-thtsw" (UID: "8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29") : failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:48.572485 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572459 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572485 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572489 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572524 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lmdk\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572549 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572582 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572620 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.572780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572763 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted\") pod \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\" (UID: \"20116020-6f5c-43c8-8fb6-d4f5c3833d5c\") " Sep 11 14:16:48.573014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.572894 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:48.573106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.573067 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:48.573161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.573107 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:48.573216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.573190 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-certificates\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.573216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.573210 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-ca-trust-extracted\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.573311 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.573223 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-trusted-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.575297 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.575248 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:16:48.575297 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.575261 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:16:48.575414 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.575343 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:48.575414 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.575401 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk" (OuterVolumeSpecName: "kube-api-access-5lmdk") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "kube-api-access-5lmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:48.575923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.575901 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "20116020-6f5c-43c8-8fb6-d4f5c3833d5c" (UID: "20116020-6f5c-43c8-8fb6-d4f5c3833d5c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:48.595204 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:48.595185 2571 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:16:48.674389 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.674362 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-bound-sa-token\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.674389 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.674385 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-registry-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.674389 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.674395 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmdk\" (UniqueName: \"kubernetes.io/projected/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-kube-api-access-5lmdk\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.674563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.674404 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-installation-pull-secrets\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.674563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.674413 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20116020-6f5c-43c8-8fb6-d4f5c3833d5c-image-registry-private-configuration\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:16:48.706141 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.706115 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:16:48.707048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.707033 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 14:16:48.716277 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:48.716261 2571 projected.go:196] Error preparing data for projected volume kube-api-access-4mhbn for pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:16:48.716332 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:16:48.716317 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn podName:8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:49.21630195 +0000 UTC m=+111.787959414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4mhbn" (UniqueName: "kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn") pod "klusterlet-addon-workmgr-5455fc8d7b-thtsw" (UID: "8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29") : failed to sync configmap cache: timed out waiting for the condition Sep 11 14:16:48.797711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:48.797688 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:16:49.076828 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.076785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:49.079281 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.079254 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-klusterlet-config\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:49.201717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.201683 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:49.278375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.278340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:49.280959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.280910 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhbn\" (UniqueName: \"kubernetes.io/projected/8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29-kube-api-access-4mhbn\") pod \"klusterlet-addon-workmgr-5455fc8d7b-thtsw\" (UID: \"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:49.282829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.282805 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:49.378953 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.378854 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rbkk7" Sep 11 14:16:49.411040 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.410978 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" podStartSLOduration=2.410957576 podStartE2EDuration="2.410957576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:48.491684922 +0000 UTC m=+111.063342407" watchObservedRunningTime="2026-09-11 14:16:49.410957576 +0000 UTC m=+111.982615047" Sep 11 14:16:49.418426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.418404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84db9b5846-s5dwb" Sep 11 14:16:49.473293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.473266 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:49.500747 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.500719 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84db9b5846-s5dwb"] Sep 11 14:16:49.511713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.511675 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84db9b5846-s5dwb"] Sep 11 14:16:49.763695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.763517 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw"] Sep 11 14:16:49.766009 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:49.765980 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc44e4c_9c6b_45a3_aa0a_a8ee17907f29.slice/crio-34f16182d39d2fc9a0c6b81869143284cd079df3eaee4d3dd75fec35dd973dae WatchSource:0}: Error finding container 34f16182d39d2fc9a0c6b81869143284cd079df3eaee4d3dd75fec35dd973dae: Status 404 returned error can't find the container with id 34f16182d39d2fc9a0c6b81869143284cd079df3eaee4d3dd75fec35dd973dae Sep 11 14:16:49.957052 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:49.957020 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20116020-6f5c-43c8-8fb6-d4f5c3833d5c" path="/var/lib/kubelet/pods/20116020-6f5c-43c8-8fb6-d4f5c3833d5c/volumes" Sep 11 14:16:50.423656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:50.423618 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" event={"ID":"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29","Type":"ContainerStarted","Data":"34f16182d39d2fc9a0c6b81869143284cd079df3eaee4d3dd75fec35dd973dae"} Sep 11 14:16:51.365179 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:51.365142 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:51.453371 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:51.453317 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:51.692278 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:51.692186 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:53.435696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:53.435607 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" event={"ID":"8bc44e4c-9c6b-45a3-aa0a-a8ee17907f29","Type":"ContainerStarted","Data":"c701bba6a84477cdee6502a234bece53c9cd262f060a7a348febbe72e6cd1a85"} Sep 11 14:16:53.436153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:53.435845 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:53.437325 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:53.437306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" Sep 11 14:16:53.456495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:53.456449 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5455fc8d7b-thtsw" podStartSLOduration=3.093170043 podStartE2EDuration="6.456436347s" totalImagesPullingTime="3.363266304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:53.455474227 +0000 UTC m=+116.027131700" watchObservedRunningTime="2026-09-11 14:16:53.456436347 +0000 UTC m=+116.028093819" Sep 11 14:16:53.639229 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:53.639197 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:54.952887 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:54.952860 2571 scope.go:117] "RemoveContainer" containerID="2c81e129c66892622086447e52e02b46474d94be8ea1fed09af1801feffa1949" Sep 11 14:16:55.444032 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.443992 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:16:55.444219 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.444186 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" event={"ID":"fe0cc6a4-21f9-401c-8b7a-c76c4fb890db","Type":"ContainerStarted","Data":"5e85a196fd56f73bf321e1843b3ac63b63cbc3e07e939e0bff642da058dab68e"} Sep 11 14:16:55.444601 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.444572 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:55.450718 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.450691 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" Sep 11 14:16:55.466701 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.466653 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-bc49w" podStartSLOduration=42.766520264 podStartE2EDuration="44.466633575s" totalImagesPullingTime="1.700113311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:55.4661675 +0000 UTC m=+118.037824974" watchObservedRunningTime="2026-09-11 14:16:55.466633575 +0000 UTC m=+118.038291038" Sep 11 14:16:55.566758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.566725 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.677473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.677422 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.783509 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.783473 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.879387 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.879347 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.980913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:55.980878 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.001430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.001378 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vhl8v"] Sep 11 14:16:56.004913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.004892 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.007762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.007742 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 14:16:56.007854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.007742 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 14:16:56.008221 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.008186 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 14:16:56.008221 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.008207 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 14:16:56.008532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.008201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 14:16:56.008532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.008291 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 14:16:56.008620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.008581 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qj5r6\"" Sep 11 14:16:56.131824 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.131740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-sys\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.131824 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.131788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxlh\" (UniqueName: \"kubernetes.io/projected/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-kube-api-access-qtxlh\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.131852 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.131903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-root\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.131924 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-textfile\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.132010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-metrics-client-ca\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.132078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.132113 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-tls\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.132149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.132132 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-wtmp\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.137576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.137516 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.213379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.213345 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.233005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.232974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-metrics-client-ca\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233164 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233031 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233164 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-tls\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233164 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-wtmp\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-sys\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxlh\" (UniqueName: \"kubernetes.io/projected/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-kube-api-access-qtxlh\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233330 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-wtmp\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233347 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-sys\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-root\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-textfile\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233523 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-root\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233693 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233766 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233709 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-metrics-client-ca\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.233766 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.233758 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-textfile\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.235483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.235466 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-tls\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.235737 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.235716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.266396 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.266362 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxlh\" (UniqueName: \"kubernetes.io/projected/2d3b1d86-eccb-45f7-9cb8-3285b8f04027-kube-api-access-qtxlh\") pod \"node-exporter-vhl8v\" (UID: \"2d3b1d86-eccb-45f7-9cb8-3285b8f04027\") " pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.308312 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.308280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.314137 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.314115 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhl8v" Sep 11 14:16:56.325525 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:16:56.325497 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3b1d86_eccb_45f7_9cb8_3285b8f04027.slice/crio-42753ec073092c3447c02a9dcb8e8872a3aa2ab94498b2f4fef6cde2e8f59f8b WatchSource:0}: Error finding container 42753ec073092c3447c02a9dcb8e8872a3aa2ab94498b2f4fef6cde2e8f59f8b: Status 404 returned error can't find the container with id 42753ec073092c3447c02a9dcb8e8872a3aa2ab94498b2f4fef6cde2e8f59f8b Sep 11 14:16:56.440488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.440406 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.447944 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.447905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhl8v" event={"ID":"2d3b1d86-eccb-45f7-9cb8-3285b8f04027","Type":"ContainerStarted","Data":"42753ec073092c3447c02a9dcb8e8872a3aa2ab94498b2f4fef6cde2e8f59f8b"} Sep 11 14:16:56.546902 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.546870 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.710507 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.710425 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.993974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:56.993861 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.077115 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.077078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.185116 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.185084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.395679 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.395645 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.453261 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.453234 2571 generic.go:356] "Generic (PLEG): container finished" podID="2d3b1d86-eccb-45f7-9cb8-3285b8f04027" containerID="f283f2cc11c91ea28cac0cb77c0e6ba7d2111e3e5cd7b9707a07e97d4c186559" exitCode=0 Sep 11 14:16:57.453374 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.453320 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhl8v" event={"ID":"2d3b1d86-eccb-45f7-9cb8-3285b8f04027","Type":"ContainerDied","Data":"f283f2cc11c91ea28cac0cb77c0e6ba7d2111e3e5cd7b9707a07e97d4c186559"} Sep 11 14:16:57.652461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:57.652432 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.093108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.093073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.294271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.294235 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.458841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.458758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhl8v" event={"ID":"2d3b1d86-eccb-45f7-9cb8-3285b8f04027","Type":"ContainerStarted","Data":"42df3306770b0e7111ce2c000b9a46192592c8bb7f09dd3bc92f0546ea80bd38"} Sep 11 14:16:58.458841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.458807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhl8v" event={"ID":"2d3b1d86-eccb-45f7-9cb8-3285b8f04027","Type":"ContainerStarted","Data":"fc24a09023c8fe90b603e89fa0162b0a1ec3d2915ed9352c24aafa980e64e767"} Sep 11 14:16:58.489323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.489282 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vhl8v" podStartSLOduration=2.552119099 podStartE2EDuration="3.489267774s" totalImagesPullingTime="877.494533ms" totalInitContainerRuntime="59.654142ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:58.48821263 +0000 UTC m=+121.059870102" watchObservedRunningTime="2026-09-11 14:16:58.489267774 +0000 UTC m=+121.060925237" Sep 11 14:16:58.715801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.715723 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.789641 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:58.789610 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.520759 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:59.520725 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.603734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:59.603697 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.705910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:59.705875 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.785267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:16:59.785239 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:00.674271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.674240 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f"] Sep 11 14:17:00.678337 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.678318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.679388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.679361 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:00.680232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.680213 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 14:17:00.680314 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.680264 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4flpz\"" Sep 11 14:17:00.704707 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.704681 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f"] Sep 11 14:17:00.765486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.765456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d611899b-7391-4314-873d-96b63b404f08-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.765631 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.765496 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4cz\" (UniqueName: \"kubernetes.io/projected/d611899b-7391-4314-873d-96b63b404f08-kube-api-access-6v4cz\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.767610 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.767582 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:00.838889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.838851 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:00.866190 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.866168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d611899b-7391-4314-873d-96b63b404f08-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.866273 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.866201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4cz\" (UniqueName: \"kubernetes.io/projected/d611899b-7391-4314-873d-96b63b404f08-kube-api-access-6v4cz\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.868504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.868482 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d611899b-7391-4314-873d-96b63b404f08-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.896292 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.896266 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4cz\" (UniqueName: \"kubernetes.io/projected/d611899b-7391-4314-873d-96b63b404f08-kube-api-access-6v4cz\") pod \"monitoring-plugin-5d85778b9d-rpf9f\" (UID: \"d611899b-7391-4314-873d-96b63b404f08\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:00.987470 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:00.987392 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:01.273214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:01.273066 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f"] Sep 11 14:17:01.275881 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:17:01.275848 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd611899b_7391_4314_873d_96b63b404f08.slice/crio-c743c47c41343ab1fe77434803e54926444ee361384012ec971d6c4114f65cce WatchSource:0}: Error finding container c743c47c41343ab1fe77434803e54926444ee361384012ec971d6c4114f65cce: Status 404 returned error can't find the container with id c743c47c41343ab1fe77434803e54926444ee361384012ec971d6c4114f65cce Sep 11 14:17:01.475720 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:01.475685 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" event={"ID":"d611899b-7391-4314-873d-96b63b404f08","Type":"ContainerStarted","Data":"c743c47c41343ab1fe77434803e54926444ee361384012ec971d6c4114f65cce"} Sep 11 14:17:02.724704 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.724660 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.725246 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.725227 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:17:02.728535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.728519 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.734527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.734501 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 14:17:02.734685 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.734635 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 14:17:02.734876 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.734861 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 14:17:02.735978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.735958 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 14:17:02.736111 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.735986 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 14:17:02.736111 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.735987 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5k66dtlfr5qj\"" Sep 11 14:17:02.736234 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736138 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 14:17:02.736234 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736190 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 14:17:02.736417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736400 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 14:17:02.736417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736400 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 14:17:02.736578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736518 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 14:17:02.736716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736699 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5whph\"" Sep 11 14:17:02.736825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.736810 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 14:17:02.739447 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.739430 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 14:17:02.739840 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.739826 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 14:17:02.740579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.740561 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 14:17:02.742005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.741992 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 14:17:02.762089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.762065 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:17:02.887926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.887896 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.887926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.887926 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.887965 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42h5\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.887989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888029 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888186 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888269 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888303 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888401 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888452 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888474 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.888783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.888558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989612 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42h5\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989846 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.989916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990182 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.989913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990314 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990447 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990650 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990627 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990669 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.990963 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.990966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991005 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991418 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.991898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.991877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.992640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.992614 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.992822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.992795 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.992899 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.992816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.992989 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.992974 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.993458 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.993540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.993870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.994170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.994402 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.994527 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.994705 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.994967 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.994924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.995204 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.995184 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.995593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.995570 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.995846 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.995827 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:02.996771 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:02.996750 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:03.001239 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.001219 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42h5\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5\") pod \"prometheus-k8s-0\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:03.038152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.038125 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:03.335058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.334974 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:17:03.337464 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:17:03.337440 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8a270_441d_46f2_8dad_6bf88c72fda7.slice/crio-70823a3b652ca4e11bfa349a9277df9122749027392b43a0ec2752ac58e9d212 WatchSource:0}: Error finding container 70823a3b652ca4e11bfa349a9277df9122749027392b43a0ec2752ac58e9d212: Status 404 returned error can't find the container with id 70823a3b652ca4e11bfa349a9277df9122749027392b43a0ec2752ac58e9d212 Sep 11 14:17:03.485244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.485208 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" event={"ID":"d611899b-7391-4314-873d-96b63b404f08","Type":"ContainerStarted","Data":"3cadfbc3a22f736052543706a2ee12d0729e2217f8a5b8ad5a062efd8bf7cecb"} Sep 11 14:17:03.485425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.485399 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:03.486614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.486588 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"70823a3b652ca4e11bfa349a9277df9122749027392b43a0ec2752ac58e9d212"} Sep 11 14:17:03.490383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.490363 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" Sep 11 14:17:03.518480 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.518441 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rpf9f" podStartSLOduration=2.03261318 podStartE2EDuration="3.51842824s" totalImagesPullingTime="1.48581506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:03.517497192 +0000 UTC m=+126.089154682" watchObservedRunningTime="2026-09-11 14:17:03.51842824 +0000 UTC m=+126.090085715" Sep 11 14:17:03.760036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.759920 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:03.862926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:03.862887 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:04.246055 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:04.246025 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:04.351113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:04.351078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:04.491734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:04.491646 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" exitCode=0 Sep 11 14:17:04.491902 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:04.491732 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} Sep 11 14:17:06.374826 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:06.374790 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:06.457043 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:06.457011 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:07.110353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.110318 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:07.226493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.226456 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:07.733098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.733077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:17:07.735178 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.735153 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/156459d2-1229-42f4-9f9b-09b858d3b4d2-metrics-certs\") pod \"network-metrics-daemon-jnjx5\" (UID: \"156459d2-1229-42f4-9f9b-09b858d3b4d2\") " pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:17:07.972125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.972051 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\"" Sep 11 14:17:07.981199 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:07.981178 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnjx5" Sep 11 14:17:08.258849 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:08.258765 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnjx5"] Sep 11 14:17:08.261486 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:17:08.261452 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156459d2_1229_42f4_9f9b_09b858d3b4d2.slice/crio-426313fa588c5b3df675396307c731f06876c9a9e0c3b97e8647f577ad7fa301 WatchSource:0}: Error finding container 426313fa588c5b3df675396307c731f06876c9a9e0c3b97e8647f577ad7fa301: Status 404 returned error can't find the container with id 426313fa588c5b3df675396307c731f06876c9a9e0c3b97e8647f577ad7fa301 Sep 11 14:17:08.510947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:08.510851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} Sep 11 14:17:08.511121 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:08.510953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} Sep 11 14:17:08.512159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:08.512123 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnjx5" event={"ID":"156459d2-1229-42f4-9f9b-09b858d3b4d2","Type":"ContainerStarted","Data":"426313fa588c5b3df675396307c731f06876c9a9e0c3b97e8647f577ad7fa301"} Sep 11 14:17:09.154230 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:09.154199 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.222836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:09.222795 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.423318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:09.423245 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f7f6bb8f9-khnkp" Sep 11 14:17:09.651479 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:09.651444 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.748825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:09.748796 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:10.525860 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.525826 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnjx5" event={"ID":"156459d2-1229-42f4-9f9b-09b858d3b4d2","Type":"ContainerStarted","Data":"1b2e02170747eba3e1c26194ab5bb7e88f04269eed33480b980ab3203ce158f8"} Sep 11 14:17:10.526320 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.525884 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnjx5" event={"ID":"156459d2-1229-42f4-9f9b-09b858d3b4d2","Type":"ContainerStarted","Data":"6aa8ffa7d6cb43ea6520692519c7ad19007c8b87c9d5f21a840f9d841426187b"} Sep 11 14:17:10.529980 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.529954 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} Sep 11 14:17:10.530151 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.530015 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} Sep 11 14:17:10.535049 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.530991 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} Sep 11 14:17:10.535049 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.531137 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerStarted","Data":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} Sep 11 14:17:10.584841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.584793 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jnjx5" podStartSLOduration=130.173961463 podStartE2EDuration="2m11.584778747s" totalImagesPullingTime="1.410817284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:10.584214213 +0000 UTC m=+133.155871686" watchObservedRunningTime="2026-09-11 14:17:10.584778747 +0000 UTC m=+133.156436219" Sep 11 14:17:10.674637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.674595 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.688449301 podStartE2EDuration="8.674580766s" totalImagesPullingTime="5.963576206s" totalInitContainerRuntime="22.555259ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:10.674281918 +0000 UTC m=+133.245939410" watchObservedRunningTime="2026-09-11 14:17:10.674580766 +0000 UTC m=+133.246238262" Sep 11 14:17:10.746200 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.746167 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:10.846717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:10.846639 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:11.747990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:11.747950 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:11.835456 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:11.835430 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:12.277838 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:12.277793 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:12.376318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:12.376283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:13.039245 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:13.039208 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:17:13.159740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:13.159703 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:13.250001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:13.249966 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.205873 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:16.205827 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.311606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:16.311576 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.407885 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:16.407856 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.851610 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:16.851574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.928342 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:16.928305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:17.756198 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:17.756163 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:17.854443 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:17.854398 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:25.224623 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:25.224584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:25.325186 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:25.325149 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:26.840004 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:26.839964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:26.964913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:26.964871 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:27.079403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:27.079365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:28.124584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:28.124547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:28.204495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:28.204459 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:29.713950 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:29.713912 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:29.814588 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:29.814552 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:30.916271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:30.916239 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:32.613235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:32.613201 2571 generic.go:356] "Generic (PLEG): container finished" podID="f1380412-5d49-4c33-b973-cf9cabe5f316" containerID="29cd1312b393c40ccb3738a01e1178aba6c7b580e78932ca7b3c9714a9285d4f" exitCode=0 Sep 11 14:17:32.613604 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:32.613277 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-g4f5j" event={"ID":"f1380412-5d49-4c33-b973-cf9cabe5f316","Type":"ContainerDied","Data":"29cd1312b393c40ccb3738a01e1178aba6c7b580e78932ca7b3c9714a9285d4f"} Sep 11 14:17:32.613646 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:32.613615 2571 scope.go:117] "RemoveContainer" containerID="29cd1312b393c40ccb3738a01e1178aba6c7b580e78932ca7b3c9714a9285d4f" Sep 11 14:17:33.619063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:33.619024 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-g4f5j" event={"ID":"f1380412-5d49-4c33-b973-cf9cabe5f316","Type":"ContainerStarted","Data":"4140171621fc6847572a18b38b8d7e9a1352152c698326336e68b0cf73646a6f"} Sep 11 14:17:34.518825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:34.518790 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:34.578921 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:34.578885 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:45.664631 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:45.664597 2571 generic.go:356] "Generic (PLEG): container finished" podID="c68b6b56-9dd9-4050-963f-39ce4edc9cbf" containerID="e509339d0e09f73294171fdabedceace7c0d59d0e40f233fb7babd2269a94d81" exitCode=0 Sep 11 14:17:45.665036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:45.664671 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" event={"ID":"c68b6b56-9dd9-4050-963f-39ce4edc9cbf","Type":"ContainerDied","Data":"e509339d0e09f73294171fdabedceace7c0d59d0e40f233fb7babd2269a94d81"} Sep 11 14:17:45.665036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:45.664967 2571 scope.go:117] "RemoveContainer" containerID="e509339d0e09f73294171fdabedceace7c0d59d0e40f233fb7babd2269a94d81" Sep 11 14:17:46.670386 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:46.670356 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tdrt5" event={"ID":"c68b6b56-9dd9-4050-963f-39ce4edc9cbf","Type":"ContainerStarted","Data":"986941998d48742352c0203b5c26ba02527083b0d34dbc48459bba2683c23298"} Sep 11 14:17:46.849669 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:46.849636 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:46.917416 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:46.917387 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:47.700950 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:17:47.700913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:01.864510 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:01.864433 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:01.964719 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:01.964675 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:03.039009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:03.038974 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:03.057734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:03.057711 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:03.746322 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:03.746297 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:12.169115 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:12.169067 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:12.264090 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:12.264053 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:15.788871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:15.788838 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:16.099078 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:16.098962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:21.289179 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289144 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:21.289667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289496 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="prometheus" containerID="cri-o://bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" gracePeriod=600 Sep 11 14:18:21.289667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289576 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="thanos-sidecar" containerID="cri-o://2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" gracePeriod=600 Sep 11 14:18:21.289769 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289714 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="config-reloader" containerID="cri-o://e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" gracePeriod=600 Sep 11 14:18:21.289815 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289718 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-web" containerID="cri-o://e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" gracePeriod=600 Sep 11 14:18:21.289863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.289852 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" gracePeriod=600 Sep 11 14:18:21.290037 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.290013 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy" containerID="cri-o://f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" gracePeriod=600 Sep 11 14:18:21.525370 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.525345 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.618438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618352 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618384 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618421 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618442 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618460 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618490 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618515 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618607 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.618695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618676 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618716 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618750 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618765 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618821 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n42h5\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618825 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618862 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619009 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.618919 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619088 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619125 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619161 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619166 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619190 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619220 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619244 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file\") pod \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\" (UID: \"e4c8a270-441d-46f2-8dad-6bf88c72fda7\") " Sep 11 14:18:21.619711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619522 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.619711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619550 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-metrics-client-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.619711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619568 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.619711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619581 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.620053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.619912 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.620345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.620271 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.620868 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.620824 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:21.621656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.621381 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.621656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.621446 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config" (OuterVolumeSpecName: "config") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.621895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.621867 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.622395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.622357 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5" (OuterVolumeSpecName: "kube-api-access-n42h5") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "kube-api-access-n42h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:21.622584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.622557 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:21.622853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.622826 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.622961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.622850 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out" (OuterVolumeSpecName: "config-out") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:21.623620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.623598 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.623823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.623804 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.624388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.624365 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:21.624542 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.624511 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.625161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.625139 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.634171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.634151 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config" (OuterVolumeSpecName: "web-config") pod "e4c8a270-441d-46f2-8dad-6bf88c72fda7" (UID: "e4c8a270-441d-46f2-8dad-6bf88c72fda7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:21.720291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720262 2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-kube-rbac-proxy\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720287 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-tls-assets\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720296 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config-out\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720304 2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-metrics-client-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720312 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720320 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42h5\" (UniqueName: \"kubernetes.io/projected/e4c8a270-441d-46f2-8dad-6bf88c72fda7-kube-api-access-n42h5\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720330 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720338 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-web-config\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720347 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720355 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720363 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-db\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720373 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720381 2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-grpc-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720389 2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720396 2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-config\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720404 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4c8a270-441d-46f2-8dad-6bf88c72fda7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.720504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.720413 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c8a270-441d-46f2-8dad-6bf88c72fda7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:18:21.803088 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803053 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" exitCode=0 Sep 11 14:18:21.803088 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803086 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" exitCode=0 Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803095 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" exitCode=0 Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803111 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" exitCode=0 Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803119 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" exitCode=0 Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803127 2571 generic.go:356] "Generic (PLEG): container finished" podID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" exitCode=0 Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803127 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803168 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803216 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803244 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} Sep 11 14:18:21.803275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} Sep 11 14:18:21.803525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803288 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.803525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803299 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} Sep 11 14:18:21.803525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} Sep 11 14:18:21.803525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.803347 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4c8a270-441d-46f2-8dad-6bf88c72fda7","Type":"ContainerDied","Data":"70823a3b652ca4e11bfa349a9277df9122749027392b43a0ec2752ac58e9d212"} Sep 11 14:18:21.811399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.811380 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.818281 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.818258 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.824888 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.824859 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.831583 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.831567 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.832838 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.832809 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:21.838021 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.838003 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.844823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.844805 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.851068 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851044 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.851368 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.851347 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.851449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851374 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.851449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851404 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.851648 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.851629 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.851711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851659 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.851711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851683 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.851923 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.851904 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.851978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851929 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.851978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.851965 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.852197 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.852181 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.852238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852199 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.852238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852211 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.852405 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.852390 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.852449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852408 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.852449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852420 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.852637 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.852621 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.852695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852644 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.852695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852664 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.852925 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:18:21.852911 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.852985 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852928 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.852985 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.852963 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.853235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853217 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.853280 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853235 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.853463 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853447 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.853527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853466 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.853790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853767 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.853842 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.853791 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.854266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854243 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.854266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854265 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.854572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854546 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.854630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854574 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.854877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854854 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.854951 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.854879 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.855232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855200 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.855232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855226 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.855514 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855492 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.855572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855517 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.855809 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855790 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.855865 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.855809 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.856089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856068 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.856089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856087 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.856336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856314 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.856388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856338 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.856574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856559 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:21.856611 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856560 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.856611 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856606 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.856823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856801 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.856823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.856823 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.857058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857042 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.857058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857057 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.857279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857263 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.857279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857277 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.857494 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857479 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.857544 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857493 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.857701 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857686 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.857739 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857701 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.857883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857868 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.857927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.857883 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.858137 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858116 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.858137 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858136 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.858335 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858317 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.858375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858336 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.858556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858538 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.858556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858556 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.858752 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858738 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.858792 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.858760 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.859065 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859044 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.859065 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859065 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.859294 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859271 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.859294 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859288 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.859483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859463 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.859531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859483 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.859672 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859656 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.859714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859673 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.859859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859842 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.859908 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.859859 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.860112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860094 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.860192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860115 2571 scope.go:117] "RemoveContainer" containerID="4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a" Sep 11 14:18:21.860346 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860329 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a"} err="failed to get container status \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": rpc error: code = NotFound desc = could not find container \"4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a\": container with ID starting with 4b708bb3624d4cb51ae8af3a7f6bde99661de328f13c945e3068bdba708dc72a not found: ID does not exist" Sep 11 14:18:21.860399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860346 2571 scope.go:117] "RemoveContainer" containerID="f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4" Sep 11 14:18:21.860566 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860550 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4"} err="failed to get container status \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": rpc error: code = NotFound desc = could not find container \"f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4\": container with ID starting with f91de363d11cc0d35f53978e5f68a73d9fc9c3ec03f04815a82c19ffec962cb4 not found: ID does not exist" Sep 11 14:18:21.860615 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860567 2571 scope.go:117] "RemoveContainer" containerID="e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58" Sep 11 14:18:21.860781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860767 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58"} err="failed to get container status \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": rpc error: code = NotFound desc = could not find container \"e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58\": container with ID starting with e976070d7ae40b4bfe8500d77cc84094f03ecf2d352d21d35b1455e8925c4d58 not found: ID does not exist" Sep 11 14:18:21.860781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.860781 2571 scope.go:117] "RemoveContainer" containerID="2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9" Sep 11 14:18:21.861045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861028 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9"} err="failed to get container status \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": rpc error: code = NotFound desc = could not find container \"2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9\": container with ID starting with 2e9cfd5e30906e971f3de48aef57752563c5e408348ccef639e771b857f42fb9 not found: ID does not exist" Sep 11 14:18:21.861120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861048 2571 scope.go:117] "RemoveContainer" containerID="e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04" Sep 11 14:18:21.861282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861263 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04"} err="failed to get container status \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": rpc error: code = NotFound desc = could not find container \"e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04\": container with ID starting with e4ad4ea8fc68923d9d05479bd72b24673f2f614448bbe7ddea973de7d459cf04 not found: ID does not exist" Sep 11 14:18:21.861329 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861284 2571 scope.go:117] "RemoveContainer" containerID="bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7" Sep 11 14:18:21.861514 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861496 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7"} err="failed to get container status \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": rpc error: code = NotFound desc = could not find container \"bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7\": container with ID starting with bfe1f6bd8d8f475c97c60cdea6bbe5ed621d78c4f65405874736a914c85a93d7 not found: ID does not exist" Sep 11 14:18:21.861572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861515 2571 scope.go:117] "RemoveContainer" containerID="c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9" Sep 11 14:18:21.861755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.861737 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9"} err="failed to get container status \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": rpc error: code = NotFound desc = could not find container \"c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9\": container with ID starting with c1791d0f390ae63724b88084cfbe9b2692879e7f37d065a22d5f51a6a9b51ac9 not found: ID does not exist" Sep 11 14:18:21.894185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894132 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:21.894394 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894381 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="config-reloader" Sep 11 14:18:21.894434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894397 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="config-reloader" Sep 11 14:18:21.894434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894409 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="init-config-reloader" Sep 11 14:18:21.894434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894414 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="init-config-reloader" Sep 11 14:18:21.894434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894421 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-web" Sep 11 14:18:21.894434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894429 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-web" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894442 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="thanos-sidecar" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894448 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="thanos-sidecar" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894460 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-thanos" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894465 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-thanos" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894473 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="prometheus" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894478 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="prometheus" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894484 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894489 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894540 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="prometheus" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894547 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894555 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="config-reloader" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894560 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="thanos-sidecar" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894565 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-web" Sep 11 14:18:21.894580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.894571 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" containerName="kube-rbac-proxy-thanos" Sep 11 14:18:21.899794 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.899778 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.908168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908151 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 14:18:21.908250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908236 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 14:18:21.908359 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908346 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 14:18:21.908493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908477 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 14:18:21.908608 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908505 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 14:18:21.908608 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908589 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 14:18:21.908713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908643 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 14:18:21.908769 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.908710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5whph\"" Sep 11 14:18:21.909102 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909089 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 14:18:21.909274 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909258 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 14:18:21.909330 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909311 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5k66dtlfr5qj\"" Sep 11 14:18:21.909388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909335 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 14:18:21.909388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909361 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 14:18:21.909545 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.909530 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 14:18:21.910025 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.910005 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 14:18:21.911648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.911634 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 14:18:21.916649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.916632 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 14:18:21.922291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922479 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922418 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922479 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922576 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922699 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922752 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922819 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.922871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.923558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922900 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.923558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.923558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.922987 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fph\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-kube-api-access-x8fph\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.923558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.923012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:21.924338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.924318 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:21.957156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:21.957131 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c8a270-441d-46f2-8dad-6bf88c72fda7" path="/var/lib/kubelet/pods/e4c8a270-441d-46f2-8dad-6bf88c72fda7/volumes" Sep 11 14:18:22.024366 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024378 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fph\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-kube-api-access-x8fph\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024666 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024854 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.024881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.024977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025005 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025298 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025366 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.025657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.025603 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.026047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.026026 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.026493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.026472 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.027991 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.027962 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.028334 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.028312 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.028711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.028688 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029101 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029079 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029181 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029086 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-config\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029324 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029513 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b29786e6-3e12-4910-a2a6-80d7b4069a82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029600 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029542 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029600 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029544 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029785 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029767 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029769 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.029859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.029842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29786e6-3e12-4910-a2a6-80d7b4069a82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.030107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.030082 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.031867 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.031849 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b29786e6-3e12-4910-a2a6-80d7b4069a82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.037949 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.037920 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fph\" (UniqueName: \"kubernetes.io/projected/b29786e6-3e12-4910-a2a6-80d7b4069a82-kube-api-access-x8fph\") pod \"prometheus-k8s-0\" (UID: \"b29786e6-3e12-4910-a2a6-80d7b4069a82\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.209231 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.209149 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:22.510625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.510601 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:18:22.512485 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:18:22.512457 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29786e6_3e12_4910_a2a6_80d7b4069a82.slice/crio-404e44404963c7ccf7435732d574c964a7e1d4ca33f66299f7ecda5896827dc0 WatchSource:0}: Error finding container 404e44404963c7ccf7435732d574c964a7e1d4ca33f66299f7ecda5896827dc0: Status 404 returned error can't find the container with id 404e44404963c7ccf7435732d574c964a7e1d4ca33f66299f7ecda5896827dc0 Sep 11 14:18:22.808630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.808599 2571 generic.go:356] "Generic (PLEG): container finished" podID="b29786e6-3e12-4910-a2a6-80d7b4069a82" containerID="adbb77a4cc0d60a99ce4bd74fb13bde5a8e06a2423df2f0580374203f32251c9" exitCode=0 Sep 11 14:18:22.808818 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.808695 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerDied","Data":"adbb77a4cc0d60a99ce4bd74fb13bde5a8e06a2423df2f0580374203f32251c9"} Sep 11 14:18:22.808818 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:22.808778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"404e44404963c7ccf7435732d574c964a7e1d4ca33f66299f7ecda5896827dc0"} Sep 11 14:18:23.564302 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.564259 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:23.817706 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817621 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"af4e0c87a83bc11e883ccbcc2968272325e76a5fcc982ee179880993c57a469f"} Sep 11 14:18:23.817843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"693db7ff851bdec9cd949dc183176201566b73bf39101ac1917cd16431d02765"} Sep 11 14:18:23.817843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817777 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"35b1f0d38f0357477038467d49790066cc79c3af1dba247cba194fd23739bf78"} Sep 11 14:18:23.817843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817820 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"fbb9f5ed6ab6a84026ec243e1202af49d7eabb52c495a005d6ca2a21b55ad7ff"} Sep 11 14:18:23.817997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817856 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"6b5222e2c1c2c81ab82f14eb94151489d6ed1c7f03cca8e4c0455c76fcc6fdd2"} Sep 11 14:18:23.817997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.817892 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b29786e6-3e12-4910-a2a6-80d7b4069a82","Type":"ContainerStarted","Data":"20eb71269144081274058cae0af89df57ef0ba4a6464f6117d526a9204babc80"} Sep 11 14:18:23.868744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.868677 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.848989737 podStartE2EDuration="2.868657466s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.667729ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:23.868487476 +0000 UTC m=+206.440144990" watchObservedRunningTime="2026-09-11 14:18:23.868657466 +0000 UTC m=+206.440314918" Sep 11 14:18:23.889120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.889083 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:23.974783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:23.974750 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:24.487305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:24.487267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:24.562854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:24.562822 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:27.209658 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:27.209626 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:18:34.258256 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:34.258212 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:34.599944 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:34.599893 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:34.738816 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:34.738779 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:34.840052 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:18:34.840018 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:02.725641 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:02.725606 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:02.852813 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:02.852775 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:22.209520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:22.209430 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:19:22.225617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:22.225589 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:19:23.049054 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:23.049026 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:19:31.187108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:31.187060 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:31.262988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:31.262948 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:33.336097 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:33.336056 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:33.446075 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:33.446036 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:38.717042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:38.717004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:38.793757 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:38.793724 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:43.024959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:43.024908 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:43.157558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:43.157527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:44.030012 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:44.029974 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:44.122874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:44.122836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:46.855004 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:46.854962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:46.960065 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:46.960037 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:47.074790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:47.074760 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:57.859516 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:57.859491 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:19:57.871237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:57.871026 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:19:57.879359 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:19:57.879327 2571 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:20:05.465215 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:05.465181 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:29.272038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:29.272003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:29.451524 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:29.451492 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:53.517132 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:53.517046 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:54.781554 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:54.781518 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:57.784663 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:57.784626 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:57.887879 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:57.887846 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:59.532663 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:59.532632 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:59.674563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:20:59.674527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:00.273392 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:00.273362 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:00.427677 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:00.427636 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:02.083874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:02.083840 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:02.273554 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:02.273523 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:03.973837 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:03.973801 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:04.079906 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:04.079873 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:09.366015 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:09.365978 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:10.915349 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:10.915316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:11.047574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:11.047536 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.771186 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:17.771152 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.882958 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:17.882908 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:19.875588 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:19.875555 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:24.361733 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:24.361692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:24.475480 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:24.475447 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:32.664053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:32.664012 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:32.923245 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:32.923155 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:33.155910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:33.155876 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:33.926495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:33.926458 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:34.112969 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.112919 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:34.204944 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.204844 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6"] Sep 11 14:21:34.208290 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.208253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.210658 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.210634 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:21:34.211195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.211180 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:21:34.211368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.211355 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srflq\"" Sep 11 14:21:34.213091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.213072 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:21:34.216665 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.216647 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:21:34.245521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.245497 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6"] Sep 11 14:21:34.320986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.320952 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.320986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.320985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.321167 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.321058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcl2\" (UniqueName: \"kubernetes.io/projected/d6d57ada-91b2-4c4c-9711-1e80b10293ef-kube-api-access-zlcl2\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.321167 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.321104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.321167 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.321124 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.421781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.421745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.421781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.421784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.422064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.421857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.422064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.421881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.422064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.421927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcl2\" (UniqueName: \"kubernetes.io/projected/d6d57ada-91b2-4c4c-9711-1e80b10293ef-kube-api-access-zlcl2\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.422282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.422260 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.423849 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.423818 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:34.424512 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.424483 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d57ada-91b2-4c4c-9711-1e80b10293ef-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.424767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.424751 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.424921 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.424901 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d57ada-91b2-4c4c-9711-1e80b10293ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.450814 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.450787 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcl2\" (UniqueName: \"kubernetes.io/projected/d6d57ada-91b2-4c4c-9711-1e80b10293ef-kube-api-access-zlcl2\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-nkjm6\" (UID: \"d6d57ada-91b2-4c4c-9711-1e80b10293ef\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.518508 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.518440 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:34.543356 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.543325 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:34.682237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.682202 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:34.819216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.819182 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6"] Sep 11 14:21:34.821717 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:21:34.821692 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d57ada_91b2_4c4c_9711_1e80b10293ef.slice/crio-1976c489cc9ee4b5c52247033a8398810cbf47704a5f108fe5aad16b6b5bd1c3 WatchSource:0}: Error finding container 1976c489cc9ee4b5c52247033a8398810cbf47704a5f108fe5aad16b6b5bd1c3: Status 404 returned error can't find the container with id 1976c489cc9ee4b5c52247033a8398810cbf47704a5f108fe5aad16b6b5bd1c3 Sep 11 14:21:34.823673 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.823657 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:21:34.837178 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:34.837146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:35.012369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:35.012331 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:35.115432 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:35.115340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:35.209778 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:35.209740 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:35.517881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:35.517803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerStarted","Data":"1976c489cc9ee4b5c52247033a8398810cbf47704a5f108fe5aad16b6b5bd1c3"} Sep 11 14:21:38.531336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:38.531304 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerStarted","Data":"df89c3ea873dc84d39fe1fe44ae7d423716c40e56e9eee12f3b728b451106dde"} Sep 11 14:21:39.537331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:39.537297 2571 generic.go:356] "Generic (PLEG): container finished" podID="d6d57ada-91b2-4c4c-9711-1e80b10293ef" containerID="df89c3ea873dc84d39fe1fe44ae7d423716c40e56e9eee12f3b728b451106dde" exitCode=0 Sep 11 14:21:39.537755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:39.537387 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerDied","Data":"df89c3ea873dc84d39fe1fe44ae7d423716c40e56e9eee12f3b728b451106dde"} Sep 11 14:21:40.543758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:40.543726 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerStarted","Data":"4c6fe43d7860a5a4e16707a6e5ef292f835aecbb6fe674e8ae4933847223c532"} Sep 11 14:21:40.544142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:40.543856 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:40.578898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:40.578849 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" podStartSLOduration=1.870005934 podStartE2EDuration="6.578833386s" totalImagesPullingTime="3.657699457s" totalInitContainerRuntime="1.051127995s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:40.577539087 +0000 UTC m=+403.149196601" watchObservedRunningTime="2026-09-11 14:21:40.578833386 +0000 UTC m=+403.150490856" Sep 11 14:21:41.800726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:41.800692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:41.984391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:41.984355 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:51.549921 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:51.549891 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:21:54.059203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:54.059155 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.162470 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:54.162433 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.397108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:54.397016 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.624729 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:54.624683 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:57.451088 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:57.451052 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:57.547528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:57.547495 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:59.246250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:21:59.246215 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:01.167384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:01.167343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:01.266323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:01.266287 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.370476 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:02.370440 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.670465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:02.670376 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.891877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:02.891844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.401689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.401653 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.491345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.491307 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.634416 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.634365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.737262 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.737182 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.993891 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.993805 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bdsmq"] Sep 11 14:22:05.998881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:05.998857 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.002334 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.002311 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 14:22:06.014015 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.013988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bdsmq"] Sep 11 14:22:06.029197 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.029167 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:06.095331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.095297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.095331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.095335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6lf\" (UniqueName: \"kubernetes.io/projected/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-kube-api-access-5j6lf\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.196801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.196763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.196801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.196806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6lf\" (UniqueName: \"kubernetes.io/projected/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-kube-api-access-5j6lf\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.197071 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:22:06.196923 2571 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 14:22:06.197071 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:22:06.197040 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs podName:bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4 nodeName:}" failed. No retries permitted until 2026-09-11 14:22:06.697016002 +0000 UTC m=+429.268673453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs") pod "model-serving-api-54d89bd5fc-bdsmq" (UID: "bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4") : secret "model-serving-api-tls" not found Sep 11 14:22:06.208484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.208453 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6lf\" (UniqueName: \"kubernetes.io/projected/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-kube-api-access-5j6lf\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.611063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.611032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:06.701184 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.701149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.703946 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.703896 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4-tls-certs\") pod \"model-serving-api-54d89bd5fc-bdsmq\" (UID: \"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4\") " pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:06.910563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:06.910459 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:07.189247 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:07.189170 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bdsmq"] Sep 11 14:22:07.192219 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:22:07.192187 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfc5a69_f7c0_40ea_bcf3_d024eb1304b4.slice/crio-2e1a41ee75ac89273be9dcb5e7502d5fb6fe8b7c67b63e40be94e13904c8192a WatchSource:0}: Error finding container 2e1a41ee75ac89273be9dcb5e7502d5fb6fe8b7c67b63e40be94e13904c8192a: Status 404 returned error can't find the container with id 2e1a41ee75ac89273be9dcb5e7502d5fb6fe8b7c67b63e40be94e13904c8192a Sep 11 14:22:07.616081 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:07.616031 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:07.647033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:07.646998 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" event={"ID":"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4","Type":"ContainerStarted","Data":"2e1a41ee75ac89273be9dcb5e7502d5fb6fe8b7c67b63e40be94e13904c8192a"} Sep 11 14:22:07.694512 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:07.694486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:10.660120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:10.660088 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" event={"ID":"bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4","Type":"ContainerStarted","Data":"ddf3f8c3d9b6666307e2ae9ff411dc638e479e44ab955c5dd70289a3fdac95b5"} Sep 11 14:22:10.660498 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:10.660200 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:10.689099 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:10.689054 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" podStartSLOduration=3.158785747 podStartE2EDuration="5.689038362s" totalImagesPullingTime="2.530252615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:10.687818566 +0000 UTC m=+433.259476059" watchObservedRunningTime="2026-09-11 14:22:10.689038362 +0000 UTC m=+433.260695835" Sep 11 14:22:12.079292 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.079254 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq"] Sep 11 14:22:12.082662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.082640 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.090637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.090614 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 14:22:12.091143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.091124 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 14:22:12.093807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.093790 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 14:22:12.094327 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.094313 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:22:12.094416 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.094318 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jr844\"" Sep 11 14:22:12.094416 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.094393 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 14:22:12.102099 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.102080 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq"] Sep 11 14:22:12.148365 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.148336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkv4x\" (UniqueName: \"kubernetes.io/projected/2903db5b-3a78-49c2-a168-16515c8b5ee9-kube-api-access-rkv4x\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.148472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.148386 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2903db5b-3a78-49c2-a168-16515c8b5ee9-manager-config\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.148472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.148403 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.148539 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.148508 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.249558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.249532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.249713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.249569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkv4x\" (UniqueName: \"kubernetes.io/projected/2903db5b-3a78-49c2-a168-16515c8b5ee9-kube-api-access-rkv4x\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.249713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.249609 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2903db5b-3a78-49c2-a168-16515c8b5ee9-manager-config\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.249713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.249633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.250380 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.250358 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2903db5b-3a78-49c2-a168-16515c8b5ee9-manager-config\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.251790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.251757 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.251977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.251956 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2903db5b-3a78-49c2-a168-16515c8b5ee9-cert\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.266564 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.266533 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkv4x\" (UniqueName: \"kubernetes.io/projected/2903db5b-3a78-49c2-a168-16515c8b5ee9-kube-api-access-rkv4x\") pod \"lws-controller-manager-744bf4ff48-n9zzq\" (UID: \"2903db5b-3a78-49c2-a168-16515c8b5ee9\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.391822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.391738 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:12.681113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:12.681031 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq"] Sep 11 14:22:12.685644 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:22:12.685619 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2903db5b_3a78_49c2_a168_16515c8b5ee9.slice/crio-0e032028c42851b4e612bd1e805208c5382bc6a250fb7a44ade60ea84caff832 WatchSource:0}: Error finding container 0e032028c42851b4e612bd1e805208c5382bc6a250fb7a44ade60ea84caff832: Status 404 returned error can't find the container with id 0e032028c42851b4e612bd1e805208c5382bc6a250fb7a44ade60ea84caff832 Sep 11 14:22:13.675491 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.675452 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" event={"ID":"2903db5b-3a78-49c2-a168-16515c8b5ee9","Type":"ContainerStarted","Data":"0e032028c42851b4e612bd1e805208c5382bc6a250fb7a44ade60ea84caff832"} Sep 11 14:22:13.754467 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.754389 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9"] Sep 11 14:22:13.758310 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.758085 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.764304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.764279 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 14:22:13.764447 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.764352 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 14:22:13.764447 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.764279 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:22:13.765387 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.765186 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 14:22:13.765387 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.765234 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:22:13.765558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.765502 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lf2rd\"" Sep 11 14:22:13.765715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.765699 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:22:13.779362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.779315 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9"] Sep 11 14:22:13.862196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862165 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862203 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbq9\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-kube-api-access-djbq9\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.862696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.862405 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963738 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963738 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963723 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbq9\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-kube-api-access-djbq9\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963798 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.963974 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.963851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.964526 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.964494 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.966987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.966955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.967108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.966955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.967570 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.967543 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.967780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.967763 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.975789 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.975758 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:13.976131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:13.976109 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbq9\" (UniqueName: \"kubernetes.io/projected/52f5f2dc-7b3d-4928-a72d-5b6df15c73e6-kube-api-access-djbq9\") pod \"istiod-openshift-gateway-5677ccb49b-4gqt9\" (UID: \"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:14.071667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:14.071630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:15.251515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:15.251489 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9"] Sep 11 14:22:15.253425 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:22:15.253395 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f5f2dc_7b3d_4928_a72d_5b6df15c73e6.slice/crio-fd71824db3c6b178999d89529c4b5d79b1fa896bc28f06f81bb9b18412a2ba8f WatchSource:0}: Error finding container fd71824db3c6b178999d89529c4b5d79b1fa896bc28f06f81bb9b18412a2ba8f: Status 404 returned error can't find the container with id fd71824db3c6b178999d89529c4b5d79b1fa896bc28f06f81bb9b18412a2ba8f Sep 11 14:22:15.684954 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:15.684904 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" event={"ID":"2903db5b-3a78-49c2-a168-16515c8b5ee9","Type":"ContainerStarted","Data":"53a557ab5afdf2fd78f8857702ef4dfac14a2abcd2c3b3efa6735eafe883e8ca"} Sep 11 14:22:15.685147 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:15.684986 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:15.686210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:15.686187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" event={"ID":"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6","Type":"ContainerStarted","Data":"fd71824db3c6b178999d89529c4b5d79b1fa896bc28f06f81bb9b18412a2ba8f"} Sep 11 14:22:15.725884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:15.725844 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" podStartSLOduration=1.3852967139999999 podStartE2EDuration="3.725831546s" totalImagesPullingTime="2.340534832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:15.7245302 +0000 UTC m=+438.296187694" watchObservedRunningTime="2026-09-11 14:22:15.725831546 +0000 UTC m=+438.297489018" Sep 11 14:22:17.725961 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:17.725906 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:22:17.726216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:17.725990 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:22:17.764347 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:17.764319 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:17.908458 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:17.908427 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.053999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.053966 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.192003 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.191967 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.357820 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.357723 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.600094 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.600053 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.702373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.702293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" event={"ID":"52f5f2dc-7b3d-4928-a72d-5b6df15c73e6","Type":"ContainerStarted","Data":"4056c13ae11cec211e752319f731c70b0f6dc0d8954d32b83456c19b36e62590"} Sep 11 14:22:18.702580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.702559 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:18.703981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.703952 2571 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-4gqt9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:22:18.704083 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.704012 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" podUID="52f5f2dc-7b3d-4928-a72d-5b6df15c73e6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:22:18.712552 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.712524 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:18.767739 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.767688 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" podStartSLOduration=3.297562198 podStartE2EDuration="5.767671822s" totalImagesPullingTime="2.470109624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:18.763771319 +0000 UTC m=+441.335428792" watchObservedRunningTime="2026-09-11 14:22:18.767671822 +0000 UTC m=+441.339329292" Sep 11 14:22:18.815640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:18.815604 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:19.363861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:19.363833 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:19.494677 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:19.494645 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:19.707343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:19.707264 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4gqt9" Sep 11 14:22:19.723597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:19.723569 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:21.668634 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:21.668604 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bdsmq" Sep 11 14:22:26.694249 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:26.694221 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-n9zzq" Sep 11 14:22:30.967369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:22:30.967334 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:06.456710 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:06.456670 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:06.568582 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:06.568524 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:15.962801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:15.962768 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:18.583576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:18.583541 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:18.667219 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:18.667181 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:19.935664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:19.935628 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:20.729629 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:20.729594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:22.764131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:22.764090 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:23.421149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:23.421116 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:24.201736 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:24.201701 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:24.781761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:24.781718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:26.340853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:26.340814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:26.456993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:26.456962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:28.528465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:28.528425 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:28.806417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:28.806330 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:28.953861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:28.953818 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:29.055392 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:29.055355 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:29.351983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:29.351951 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:29.454992 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:29.454953 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:34.559687 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:34.559649 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:34.657196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:34.657165 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:34.986555 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:34.986475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:35.133852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:35.133814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:38.788193 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:38.788159 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:38.885851 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:38.885816 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.983110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:42.983066 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:43.129148 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:43.129111 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:45.971379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:45.971340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:46.230427 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:46.230348 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:47.619573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:47.619541 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:47.724928 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:47.724897 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:47.922553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:47.922476 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:55.712957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:55.712855 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:55.862052 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:55.862016 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:56.317318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:56.317281 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:56.423854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:23:56.423819 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:00.070854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:00.070814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:00.180387 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:00.180351 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:04.084808 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:04.084762 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:04.194902 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:04.194858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:16.177515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:16.177481 2571 generic.go:356] "Generic (PLEG): container finished" podID="d6d57ada-91b2-4c4c-9711-1e80b10293ef" containerID="4c6fe43d7860a5a4e16707a6e5ef292f835aecbb6fe674e8ae4933847223c532" exitCode=1 Sep 11 14:24:16.177927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:16.177552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerDied","Data":"4c6fe43d7860a5a4e16707a6e5ef292f835aecbb6fe674e8ae4933847223c532"} Sep 11 14:24:16.177927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:16.177900 2571 scope.go:117] "RemoveContainer" containerID="4c6fe43d7860a5a4e16707a6e5ef292f835aecbb6fe674e8ae4933847223c532" Sep 11 14:24:17.184553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:17.184523 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" event={"ID":"d6d57ada-91b2-4c4c-9711-1e80b10293ef","Type":"ContainerStarted","Data":"1226e1b1f7a58d7ebcb1ffbd5f7369e3b367ad2000d8f11c9709ac2dcfe8a1b0"} Sep 11 14:24:17.184901 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:17.184723 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:24:19.915908 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:24:19.915868 2571 controller.go:195] "Failed to update lease" err="Put \"https://a75f141b2f27e4f3f980fa6204891153-e1e569fb23f34ba2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-198.ec2.internal?timeout=10s\": context deadline exceeded" Sep 11 14:24:20.294741 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:24:20.294708 2571 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-141-198.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Sep 11 14:24:25.932758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:25.932710 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:26.048902 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:26.048863 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:28.191922 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:28.191892 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-nkjm6" Sep 11 14:24:44.736871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:44.736829 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:50.270968 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:50.270895 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:50.414250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:50.414210 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:50.521475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:50.521401 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:50.826725 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:50.826693 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:51.327593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:51.327556 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:57.895896 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:57.895871 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:24:57.898551 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:24:57.898527 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:25:08.407654 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:08.407610 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:08.702758 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:08.702673 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:23.427800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.427716 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:23.448788 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.448756 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:23.452019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.452005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.454384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.454356 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ftgll\"" Sep 11 14:25:23.454707 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.454686 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 14:25:23.454979 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.454964 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:25:23.455079 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.454974 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:25:23.467576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.467558 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:23.508664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.508627 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.508828 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.508755 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg4h\" (UniqueName: \"kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.531495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.531464 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:23.556326 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.556300 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:23.609592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.609563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg4h\" (UniqueName: \"kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.609736 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.609639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.610260 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.610242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.622002 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.621980 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg4h\" (UniqueName: \"kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h\") pod \"limitador-limitador-8588749977-btxtr\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:23.636041 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.636003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:23.762842 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:23.762767 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:24.046577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:24.046550 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:24.048914 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:25:24.048882 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40934ba0_2d2e_464f_ad6f_5543a5cbae81.slice/crio-bd12ab1ee63579c73b0d1a444c7f0b2806ea6f326d4033689f893032359dbdd8 WatchSource:0}: Error finding container bd12ab1ee63579c73b0d1a444c7f0b2806ea6f326d4033689f893032359dbdd8: Status 404 returned error can't find the container with id bd12ab1ee63579c73b0d1a444c7f0b2806ea6f326d4033689f893032359dbdd8 Sep 11 14:25:24.439459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:24.439373 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:24.463828 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:24.463803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" event={"ID":"40934ba0-2d2e-464f-ad6f-5543a5cbae81","Type":"ContainerStarted","Data":"bd12ab1ee63579c73b0d1a444c7f0b2806ea6f326d4033689f893032359dbdd8"} Sep 11 14:25:24.541442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:24.541408 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:27.478776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:27.478742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" event={"ID":"40934ba0-2d2e-464f-ad6f-5543a5cbae81","Type":"ContainerStarted","Data":"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70"} Sep 11 14:25:27.479116 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:27.478801 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:27.535807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:27.535761 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" podStartSLOduration=1.184574542 podStartE2EDuration="4.535745377s" totalImagesPullingTime="3.351170835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:27.534803477 +0000 UTC m=+630.106460965" watchObservedRunningTime="2026-09-11 14:25:27.535745377 +0000 UTC m=+630.107402871" Sep 11 14:25:37.518734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:37.518702 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:37.519172 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:37.518889 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" podUID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" containerName="limitador" containerID="cri-o://73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70" gracePeriod=30 Sep 11 14:25:37.519578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:37.519560 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:38.062019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.061996 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:38.132782 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.132713 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcg4h\" (UniqueName: \"kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h\") pod \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " Sep 11 14:25:38.132918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.132785 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file\") pod \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\" (UID: \"40934ba0-2d2e-464f-ad6f-5543a5cbae81\") " Sep 11 14:25:38.133177 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.133152 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file" (OuterVolumeSpecName: "config-file") pod "40934ba0-2d2e-464f-ad6f-5543a5cbae81" (UID: "40934ba0-2d2e-464f-ad6f-5543a5cbae81"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:25:38.134980 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.134959 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h" (OuterVolumeSpecName: "kube-api-access-qcg4h") pod "40934ba0-2d2e-464f-ad6f-5543a5cbae81" (UID: "40934ba0-2d2e-464f-ad6f-5543a5cbae81"). InnerVolumeSpecName "kube-api-access-qcg4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:38.234255 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.234223 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40934ba0-2d2e-464f-ad6f-5543a5cbae81-config-file\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:25:38.234255 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.234250 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcg4h\" (UniqueName: \"kubernetes.io/projected/40934ba0-2d2e-464f-ad6f-5543a5cbae81-kube-api-access-qcg4h\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:25:38.529391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.529361 2571 generic.go:356] "Generic (PLEG): container finished" podID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" containerID="73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70" exitCode=0 Sep 11 14:25:38.529800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.529403 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" Sep 11 14:25:38.529800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.529415 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" event={"ID":"40934ba0-2d2e-464f-ad6f-5543a5cbae81","Type":"ContainerDied","Data":"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70"} Sep 11 14:25:38.529800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.529454 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-btxtr" event={"ID":"40934ba0-2d2e-464f-ad6f-5543a5cbae81","Type":"ContainerDied","Data":"bd12ab1ee63579c73b0d1a444c7f0b2806ea6f326d4033689f893032359dbdd8"} Sep 11 14:25:38.529800 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.529475 2571 scope.go:117] "RemoveContainer" containerID="73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70" Sep 11 14:25:38.537671 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.537653 2571 scope.go:117] "RemoveContainer" containerID="73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70" Sep 11 14:25:38.537944 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:25:38.537914 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70\": container with ID starting with 73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70 not found: ID does not exist" containerID="73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70" Sep 11 14:25:38.538010 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.537966 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70"} err="failed to get container status \"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70\": rpc error: code = NotFound desc = could not find container \"73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70\": container with ID starting with 73111ad8b80d9d625b99ae10e50d7160a5979bb89d7e4f48b6cc0f0e0cf15c70 not found: ID does not exist" Sep 11 14:25:38.553607 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.553588 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:38.570170 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:38.570151 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-btxtr"] Sep 11 14:25:39.960085 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:39.960053 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" path="/var/lib/kubelet/pods/40934ba0-2d2e-464f-ad6f-5543a5cbae81/volumes" Sep 11 14:25:41.512766 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:41.512726 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:42.759957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:42.759909 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:42.865110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:42.865080 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:43.253431 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.253402 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb"] Sep 11 14:25:43.253709 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.253697 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" containerName="limitador" Sep 11 14:25:43.253753 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.253711 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" containerName="limitador" Sep 11 14:25:43.253787 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.253776 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="40934ba0-2d2e-464f-ad6f-5543a5cbae81" containerName="limitador" Sep 11 14:25:43.258024 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.258006 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.274269 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.274247 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:25:43.274397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.274330 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rv9s4\"" Sep 11 14:25:43.274397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.274343 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:25:43.274639 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.274625 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 14:25:43.287119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.287092 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb"] Sep 11 14:25:43.301761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.301740 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:43.375310 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.375276 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-data\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.375439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.375333 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45bf\" (UniqueName: \"kubernetes.io/projected/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-kube-api-access-p45bf\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.476760 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.476731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-data\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.476893 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.476789 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45bf\" (UniqueName: \"kubernetes.io/projected/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-kube-api-access-p45bf\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.477163 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.477144 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-data\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.490136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.490108 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45bf\" (UniqueName: \"kubernetes.io/projected/cb5c02c2-9f96-4b6b-93ac-fa420330bfdb-kube-api-access-p45bf\") pod \"postgres-69fb4594cf-bwsjb\" (UID: \"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.567909 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.567881 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:43.843475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:43.843401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb"] Sep 11 14:25:43.846232 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:25:43.846205 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5c02c2_9f96_4b6b_93ac_fa420330bfdb.slice/crio-a27e8e2c2d1eabb8d62f479147b3caf3a85c19c79281f6c1a4bcc0f4f2632f51 WatchSource:0}: Error finding container a27e8e2c2d1eabb8d62f479147b3caf3a85c19c79281f6c1a4bcc0f4f2632f51: Status 404 returned error can't find the container with id a27e8e2c2d1eabb8d62f479147b3caf3a85c19c79281f6c1a4bcc0f4f2632f51 Sep 11 14:25:44.557368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:44.557338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" event={"ID":"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb","Type":"ContainerStarted","Data":"a27e8e2c2d1eabb8d62f479147b3caf3a85c19c79281f6c1a4bcc0f4f2632f51"} Sep 11 14:25:45.948471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:45.948421 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:49.585119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:49.585082 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" event={"ID":"cb5c02c2-9f96-4b6b-93ac-fa420330bfdb","Type":"ContainerStarted","Data":"8833c63ba14bde3e9080c4539795f043460096ec8e305082442e21437938e6e7"} Sep 11 14:25:49.585483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:49.585157 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:49.613228 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:49.613171 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" podStartSLOduration=1.222460994 podStartE2EDuration="6.613157567s" totalImagesPullingTime="5.390696573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:49.610517591 +0000 UTC m=+652.182175079" watchObservedRunningTime="2026-09-11 14:25:49.613157567 +0000 UTC m=+652.184815041" Sep 11 14:25:55.618776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:55.618747 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bwsjb" Sep 11 14:25:55.990474 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:55.990370 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:58.547835 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:25:58.547801 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:03.896351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.896306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:03.971515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.971482 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:03.996428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.996393 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:03.997545 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.997526 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:03.997642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.997603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:03.999895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.999873 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 14:26:04.000026 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:03.999928 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 14:26:04.000222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.000204 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jhxc4\"" Sep 11 14:26:04.000435 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.000419 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 14:26:04.043591 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.043544 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.043765 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.043644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.043765 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.043675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphfq\" (UniqueName: \"kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.068362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.068332 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:04.144507 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.144483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.144607 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.144514 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphfq\" (UniqueName: \"kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.144607 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.144557 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.144689 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:26:04.144654 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 14:26:04.144726 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:26:04.144719 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert podName:045bd86b-0a11-4772-abd2-897b26125291 nodeName:}" failed. No retries permitted until 2026-09-11 14:26:04.644704651 +0000 UTC m=+667.216362101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert") pod "maas-controller-699f64d9d6-mzf8b" (UID: "045bd86b-0a11-4772-abd2-897b26125291") : secret "maas-controller-webhook-cert" not found Sep 11 14:26:04.146889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.146844 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.155715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.155693 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphfq\" (UniqueName: \"kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.168307 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.168285 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:26:04.171617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.171603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.181721 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.181701 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:26:04.244944 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.244914 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.245081 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.244989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.245081 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.245034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpf2x\" (UniqueName: \"kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.308364 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.308339 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:04.345697 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.345669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.345801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.345718 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.345801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.345748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpf2x\" (UniqueName: \"kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.348165 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.348145 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.348664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.348646 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.356318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.356293 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpf2x\" (UniqueName: \"kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x\") pod \"maas-controller-77989fd775-lknmd\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.482159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.482070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:04.648327 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.648297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.650640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.650617 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") pod \"maas-controller-699f64d9d6-mzf8b\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:04.761170 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.761144 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:26:04.762991 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:26:04.762969 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fab5aa_9618_46bf_baa1_4885dedf45b9.slice/crio-44d834dbc0cc528c23fc31499889ca345b4922c669992e371602fa7dabd326d6 WatchSource:0}: Error finding container 44d834dbc0cc528c23fc31499889ca345b4922c669992e371602fa7dabd326d6: Status 404 returned error can't find the container with id 44d834dbc0cc528c23fc31499889ca345b4922c669992e371602fa7dabd326d6 Sep 11 14:26:04.908154 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:04.908129 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:05.179772 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:05.179750 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:05.182201 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:26:05.182173 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045bd86b_0a11_4772_abd2_897b26125291.slice/crio-133df71b5fd4ca20c91d9f914c298b385b7545c862eaf29b6984a0fd97941e26 WatchSource:0}: Error finding container 133df71b5fd4ca20c91d9f914c298b385b7545c862eaf29b6984a0fd97941e26: Status 404 returned error can't find the container with id 133df71b5fd4ca20c91d9f914c298b385b7545c862eaf29b6984a0fd97941e26 Sep 11 14:26:05.658988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:05.658722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-lknmd" event={"ID":"e8fab5aa-9618-46bf-baa1-4885dedf45b9","Type":"ContainerStarted","Data":"44d834dbc0cc528c23fc31499889ca345b4922c669992e371602fa7dabd326d6"} Sep 11 14:26:05.660352 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:05.660323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" event={"ID":"045bd86b-0a11-4772-abd2-897b26125291","Type":"ContainerStarted","Data":"133df71b5fd4ca20c91d9f914c298b385b7545c862eaf29b6984a0fd97941e26"} Sep 11 14:26:07.583166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.583119 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:07.675704 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.675593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-lknmd" event={"ID":"e8fab5aa-9618-46bf-baa1-4885dedf45b9","Type":"ContainerStarted","Data":"0f36e5254c3126304352f0c1ad077251a6b097cd586f2749cbb193d7a3741cc4"} Sep 11 14:26:07.675863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.675719 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:07.677405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.677374 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:07.677673 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.677652 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" event={"ID":"045bd86b-0a11-4772-abd2-897b26125291","Type":"ContainerStarted","Data":"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665"} Sep 11 14:26:07.677749 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.677706 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:07.698054 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.698011 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77989fd775-lknmd" podStartSLOduration=1.080166096 podStartE2EDuration="3.697998023s" totalImagesPullingTime="2.617831927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:07.697327562 +0000 UTC m=+670.268985035" watchObservedRunningTime="2026-09-11 14:26:07.697998023 +0000 UTC m=+670.269655497" Sep 11 14:26:07.721063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.721018 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" podStartSLOduration=2.512399326 podStartE2EDuration="4.721002456s" totalImagesPullingTime="2.20860313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:07.71988165 +0000 UTC m=+670.291539125" watchObservedRunningTime="2026-09-11 14:26:07.721002456 +0000 UTC m=+670.292659928" Sep 11 14:26:07.879768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.879736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:07.984779 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:07.984704 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:18.684469 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:18.684435 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:26:18.684912 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:18.684501 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:18.782227 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:18.782194 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:18.782439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:18.782405 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" podUID="045bd86b-0a11-4772-abd2-897b26125291" containerName="manager" containerID="cri-o://2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665" gracePeriod=30 Sep 11 14:26:19.025926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.025898 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:19.080153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.080124 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs\") pod \"045bd86b-0a11-4772-abd2-897b26125291\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " Sep 11 14:26:19.080153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.080155 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gphfq\" (UniqueName: \"kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq\") pod \"045bd86b-0a11-4772-abd2-897b26125291\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " Sep 11 14:26:19.080316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.080208 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") pod \"045bd86b-0a11-4772-abd2-897b26125291\" (UID: \"045bd86b-0a11-4772-abd2-897b26125291\") " Sep 11 14:26:19.082329 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.082294 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "045bd86b-0a11-4772-abd2-897b26125291" (UID: "045bd86b-0a11-4772-abd2-897b26125291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:26:19.082448 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.082384 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq" (OuterVolumeSpecName: "kube-api-access-gphfq") pod "045bd86b-0a11-4772-abd2-897b26125291" (UID: "045bd86b-0a11-4772-abd2-897b26125291"). InnerVolumeSpecName "kube-api-access-gphfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:19.082506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.082442 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "045bd86b-0a11-4772-abd2-897b26125291" (UID: "045bd86b-0a11-4772-abd2-897b26125291"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:26:19.181620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.181594 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:26:19.181620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.181617 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gphfq\" (UniqueName: \"kubernetes.io/projected/045bd86b-0a11-4772-abd2-897b26125291-kube-api-access-gphfq\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:26:19.181760 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.181633 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/045bd86b-0a11-4772-abd2-897b26125291-webhook-cert\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:26:19.732542 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.732513 2571 generic.go:356] "Generic (PLEG): container finished" podID="045bd86b-0a11-4772-abd2-897b26125291" containerID="2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665" exitCode=0 Sep 11 14:26:19.732987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.732549 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" event={"ID":"045bd86b-0a11-4772-abd2-897b26125291","Type":"ContainerDied","Data":"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665"} Sep 11 14:26:19.732987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.732559 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" Sep 11 14:26:19.732987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.732626 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-699f64d9d6-mzf8b" event={"ID":"045bd86b-0a11-4772-abd2-897b26125291","Type":"ContainerDied","Data":"133df71b5fd4ca20c91d9f914c298b385b7545c862eaf29b6984a0fd97941e26"} Sep 11 14:26:19.732987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.732649 2571 scope.go:117] "RemoveContainer" containerID="2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665" Sep 11 14:26:19.741571 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.741556 2571 scope.go:117] "RemoveContainer" containerID="2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665" Sep 11 14:26:19.741797 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:26:19.741780 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665\": container with ID starting with 2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665 not found: ID does not exist" containerID="2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665" Sep 11 14:26:19.741842 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.741804 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665"} err="failed to get container status \"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665\": rpc error: code = NotFound desc = could not find container \"2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665\": container with ID starting with 2365c3f44ebe4c719f70616fd0f85ecd292855c25ca042ed88e0e738c37fb665 not found: ID does not exist" Sep 11 14:26:19.755593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.755573 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:19.762824 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.762804 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-699f64d9d6-mzf8b"] Sep 11 14:26:19.958418 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:19.958387 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="045bd86b-0a11-4772-abd2-897b26125291" path="/var/lib/kubelet/pods/045bd86b-0a11-4772-abd2-897b26125291/volumes" Sep 11 14:26:40.359231 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:40.359199 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:40.482912 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:40.482863 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:41.857664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:41.857638 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:41.966836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:41.966807 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:42.076383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.076351 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:42.154549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.154483 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:26:42.154794 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.154783 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="045bd86b-0a11-4772-abd2-897b26125291" containerName="manager" Sep 11 14:26:42.154836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.154796 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="045bd86b-0a11-4772-abd2-897b26125291" containerName="manager" Sep 11 14:26:42.154874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.154867 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="045bd86b-0a11-4772-abd2-897b26125291" containerName="manager" Sep 11 14:26:42.160973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.160951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.165622 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.165587 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:26:42.165978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.165624 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 14:26:42.166118 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.165633 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:26:42.191094 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.191067 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:26:42.246413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.246365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:42.269372 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.269348 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc69\" (UniqueName: \"kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.269503 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.269384 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.269503 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.269404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.269503 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.269425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.369895 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.369862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.370063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.369974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc69\" (UniqueName: \"kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.370063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.370001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.370063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.370018 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.370592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.370571 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.372523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.372501 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.372618 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.372569 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.379462 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.379438 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc69\" (UniqueName: \"kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69\") pod \"maas-api-67df577d97-rkdch\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.473435 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.473365 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:42.762630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.762561 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:26:42.765840 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:26:42.765815 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be5fdc_0e9c_46c5_80c1_0dd9c279c989.slice/crio-aaf900ecb02d16b35d2afcafdab55e789e578e23ef27365d035cb129dfada836 WatchSource:0}: Error finding container aaf900ecb02d16b35d2afcafdab55e789e578e23ef27365d035cb129dfada836: Status 404 returned error can't find the container with id aaf900ecb02d16b35d2afcafdab55e789e578e23ef27365d035cb129dfada836 Sep 11 14:26:42.767731 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.767712 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:26:42.835237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:42.835197 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" event={"ID":"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989","Type":"ContainerStarted","Data":"aaf900ecb02d16b35d2afcafdab55e789e578e23ef27365d035cb129dfada836"} Sep 11 14:26:43.007886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.007846 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:43.159221 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.159184 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:43.412226 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.412150 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6"] Sep 11 14:26:43.416108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.416085 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.418574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.418517 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:26:43.418695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.418603 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sf8xp\"" Sep 11 14:26:43.438790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.438762 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6"] Sep 11 14:26:43.478073 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.478039 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81aae6fa-1207-40cb-aef6-f7c32221ec45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.478245 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.478127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlpg\" (UniqueName: \"kubernetes.io/projected/81aae6fa-1207-40cb-aef6-f7c32221ec45-kube-api-access-cwlpg\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.578626 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.578593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlpg\" (UniqueName: \"kubernetes.io/projected/81aae6fa-1207-40cb-aef6-f7c32221ec45-kube-api-access-cwlpg\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.578785 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.578676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81aae6fa-1207-40cb-aef6-f7c32221ec45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.579425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.579405 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81aae6fa-1207-40cb-aef6-f7c32221ec45-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.602022 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.601992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlpg\" (UniqueName: \"kubernetes.io/projected/81aae6fa-1207-40cb-aef6-f7c32221ec45-kube-api-access-cwlpg\") pod \"payload-pre-processing-59fbc47bf8-7xwq6\" (UID: \"81aae6fa-1207-40cb-aef6-f7c32221ec45\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:43.729724 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:43.729648 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:44.028008 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:44.027981 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6"] Sep 11 14:26:44.030677 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:26:44.030645 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aae6fa_1207_40cb_aef6_f7c32221ec45.slice/crio-fe1d3b6bcc4b647a8d3341dfb27d20b1ff77c87b753772d8b3d07f6e62d1110a WatchSource:0}: Error finding container fe1d3b6bcc4b647a8d3341dfb27d20b1ff77c87b753772d8b3d07f6e62d1110a: Status 404 returned error can't find the container with id fe1d3b6bcc4b647a8d3341dfb27d20b1ff77c87b753772d8b3d07f6e62d1110a Sep 11 14:26:44.849168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:44.849129 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" event={"ID":"81aae6fa-1207-40cb-aef6-f7c32221ec45","Type":"ContainerStarted","Data":"fe1d3b6bcc4b647a8d3341dfb27d20b1ff77c87b753772d8b3d07f6e62d1110a"} Sep 11 14:26:44.851796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:44.851741 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" event={"ID":"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989","Type":"ContainerStarted","Data":"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4"} Sep 11 14:26:44.851979 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:44.851882 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:44.883363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:44.883307 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" podStartSLOduration=0.978558361 podStartE2EDuration="2.883288422s" totalImagesPullingTime="1.904730061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:44.881864008 +0000 UTC m=+707.453521505" watchObservedRunningTime="2026-09-11 14:26:44.883288422 +0000 UTC m=+707.454945895" Sep 11 14:26:45.299737 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:45.299705 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.382065 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:45.382023 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:46.864712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:46.864670 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" event={"ID":"81aae6fa-1207-40cb-aef6-f7c32221ec45","Type":"ContainerStarted","Data":"340495b7ebffdd954dfd8e0fe8cb4279439108040fa02c797559e98fc9ee1432"} Sep 11 14:26:46.865132 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:46.864806 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:26:46.905339 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:46.904959 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" podStartSLOduration=1.473215024 podStartE2EDuration="3.904928503s" totalImagesPullingTime="2.431713479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:46.90470312 +0000 UTC m=+709.476360612" watchObservedRunningTime="2026-09-11 14:26:46.904928503 +0000 UTC m=+709.476585969" Sep 11 14:26:50.864397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:50.864324 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:26:52.153344 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:52.153298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:52.251510 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:52.251474 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:54.447585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:54.447549 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:57.871573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:26:57.871544 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-7xwq6" Sep 11 14:27:13.740463 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.740427 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs"] Sep 11 14:27:13.745948 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.745912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.748240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.748216 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:27:13.748346 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.748259 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m8wqn\"" Sep 11 14:27:13.748543 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.748526 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:27:13.748857 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.748843 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 14:27:13.766303 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.766270 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:13.766636 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.766617 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs"] Sep 11 14:27:13.813461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.813602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.813602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.813602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813595 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhq8p\" (UniqueName: \"kubernetes.io/projected/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-kube-api-access-qhq8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.813714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.813714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.813656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914557 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914571 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhq8p\" (UniqueName: \"kubernetes.io/projected/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-kube-api-access-qhq8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914618 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.914716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.914679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.915050 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.915013 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.915156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.915050 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.915156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.915098 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.916990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.916974 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.917128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.917111 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:13.925265 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:13.925245 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhq8p\" (UniqueName: \"kubernetes.io/projected/e9f8a2a5-a5bd-4d2a-b147-09e8a314c315-kube-api-access-qhq8p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs\" (UID: \"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:14.056460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:14.056428 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:14.332350 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:14.332324 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs"] Sep 11 14:27:14.334428 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:27:14.334401 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f8a2a5_a5bd_4d2a_b147_09e8a314c315.slice/crio-efc4d63f086bc4b20873d4c800ea474dbe595f82a28b57e4b1f3855b91bb60e1 WatchSource:0}: Error finding container efc4d63f086bc4b20873d4c800ea474dbe595f82a28b57e4b1f3855b91bb60e1: Status 404 returned error can't find the container with id efc4d63f086bc4b20873d4c800ea474dbe595f82a28b57e4b1f3855b91bb60e1 Sep 11 14:27:14.992640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:14.992607 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" event={"ID":"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315","Type":"ContainerStarted","Data":"efc4d63f086bc4b20873d4c800ea474dbe595f82a28b57e4b1f3855b91bb60e1"} Sep 11 14:27:15.997973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:15.997926 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" event={"ID":"e9f8a2a5-a5bd-4d2a-b147-09e8a314c315","Type":"ContainerStarted","Data":"c75d0489cfaa39eb7fcb968657bf9e5dbd80761e8d78ae2d9b3a950da3545b74"} Sep 11 14:27:24.056777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:24.056740 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:24.057305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:24.056810 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:24.069331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:24.069307 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:24.096775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:24.096730 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" podStartSLOduration=10.140048441 podStartE2EDuration="11.096715348s" totalImagesPullingTime="956.666907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:16.038133176 +0000 UTC m=+738.609790660" watchObservedRunningTime="2026-09-11 14:27:24.096715348 +0000 UTC m=+746.668372821" Sep 11 14:27:25.048099 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:25.048070 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs" Sep 11 14:27:29.168371 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:29.168337 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:40.789468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:40.789432 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:43.646987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:43.646925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:57.119073 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.119035 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz"] Sep 11 14:27:57.122306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.122288 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.124639 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.124615 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 14:27:57.142406 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.142380 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz"] Sep 11 14:27:57.163829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.163799 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:57.187892 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.187863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.188048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.187925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.188048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.187983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6drt\" (UniqueName: \"kubernetes.io/projected/c38c1412-0037-4ce8-b791-56fb3a1209ee-kube-api-access-l6drt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.188048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.188008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.188160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.188074 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38c1412-0037-4ce8-b791-56fb3a1209ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.188160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.188109 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289343 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6drt\" (UniqueName: \"kubernetes.io/projected/c38c1412-0037-4ce8-b791-56fb3a1209ee-kube-api-access-l6drt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38c1412-0037-4ce8-b791-56fb3a1209ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289799 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.289928 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.289865 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.291496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.291478 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38c1412-0037-4ce8-b791-56fb3a1209ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.291845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.291829 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38c1412-0037-4ce8-b791-56fb3a1209ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.299191 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.299170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6drt\" (UniqueName: \"kubernetes.io/projected/c38c1412-0037-4ce8-b791-56fb3a1209ee-kube-api-access-l6drt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mxkqz\" (UID: \"c38c1412-0037-4ce8-b791-56fb3a1209ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.432658 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.432574 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:27:57.706398 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:57.706377 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz"] Sep 11 14:27:57.708851 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:27:57.708822 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38c1412_0037_4ce8_b791_56fb3a1209ee.slice/crio-38d4f5f2114133b14066f8276d356464343b7067420967909d02c9e51474a75a WatchSource:0}: Error finding container 38d4f5f2114133b14066f8276d356464343b7067420967909d02c9e51474a75a: Status 404 returned error can't find the container with id 38d4f5f2114133b14066f8276d356464343b7067420967909d02c9e51474a75a Sep 11 14:27:58.181456 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:58.181424 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" event={"ID":"c38c1412-0037-4ce8-b791-56fb3a1209ee","Type":"ContainerStarted","Data":"cbfc5b3da51f2c67f1556236deaa75d408a15eb72b5b64a8e14dfa784c2e6658"} Sep 11 14:27:58.181816 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:58.181478 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" event={"ID":"c38c1412-0037-4ce8-b791-56fb3a1209ee","Type":"ContainerStarted","Data":"38d4f5f2114133b14066f8276d356464343b7067420967909d02c9e51474a75a"} Sep 11 14:27:58.202731 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:27:58.202683 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" podStartSLOduration=0.96914348 podStartE2EDuration="1.202668316s" totalImagesPullingTime="233.524836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:58.202233112 +0000 UTC m=+780.773890584" watchObservedRunningTime="2026-09-11 14:27:58.202668316 +0000 UTC m=+780.774325789" Sep 11 14:28:01.719473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:01.719438 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:07.433313 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:07.433278 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:28:07.433773 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:07.433328 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:28:07.445459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:07.445433 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:28:08.237027 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:08.236997 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mxkqz" Sep 11 14:28:12.072429 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.072392 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz"] Sep 11 14:28:12.075869 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.075853 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.078222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.078201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 14:28:12.099043 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.099000 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz"] Sep 11 14:28:12.219300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219265 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.219441 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219307 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.219441 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.219535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjpk\" (UniqueName: \"kubernetes.io/projected/ea40923a-aea4-4d89-94f1-ab69c84e36f4-kube-api-access-ffjpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.219535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219473 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.219617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.219558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.240360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.240330 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:12.320727 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjpk\" (UniqueName: \"kubernetes.io/projected/ea40923a-aea4-4d89-94f1-ab69c84e36f4-kube-api-access-ffjpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.320727 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.320988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320776 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.320988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.320988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.320988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.320840 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.321279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.321257 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.321363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.321297 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.321363 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.321349 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.323077 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.323020 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea40923a-aea4-4d89-94f1-ab69c84e36f4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.323355 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.323336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea40923a-aea4-4d89-94f1-ab69c84e36f4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.334567 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.334546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjpk\" (UniqueName: \"kubernetes.io/projected/ea40923a-aea4-4d89-94f1-ab69c84e36f4-kube-api-access-ffjpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jr8sz\" (UID: \"ea40923a-aea4-4d89-94f1-ab69c84e36f4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.386720 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.386683 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:12.661521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:12.661335 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz"] Sep 11 14:28:12.664200 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:28:12.664174 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea40923a_aea4_4d89_94f1_ab69c84e36f4.slice/crio-3408e7423e561c928c7ff6eb9561642349c1eca49196e4e9ae418fba1ac4dc7a WatchSource:0}: Error finding container 3408e7423e561c928c7ff6eb9561642349c1eca49196e4e9ae418fba1ac4dc7a: Status 404 returned error can't find the container with id 3408e7423e561c928c7ff6eb9561642349c1eca49196e4e9ae418fba1ac4dc7a Sep 11 14:28:13.249048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:13.248920 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" event={"ID":"ea40923a-aea4-4d89-94f1-ab69c84e36f4","Type":"ContainerStarted","Data":"f0afc5018bcb2022255cbb11e82d7f52945c138119c436e21c894e45cd0c4555"} Sep 11 14:28:13.249048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:13.249009 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" event={"ID":"ea40923a-aea4-4d89-94f1-ab69c84e36f4","Type":"ContainerStarted","Data":"3408e7423e561c928c7ff6eb9561642349c1eca49196e4e9ae418fba1ac4dc7a"} Sep 11 14:28:13.275251 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:13.275194 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" podStartSLOduration=1.076119933 podStartE2EDuration="1.275174727s" totalImagesPullingTime="199.054794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:13.273223764 +0000 UTC m=+795.844881258" watchObservedRunningTime="2026-09-11 14:28:13.275174727 +0000 UTC m=+795.846832200" Sep 11 14:28:22.387845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:22.387745 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:22.387845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:22.387819 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:22.401052 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:22.401024 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:23.303658 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:23.303630 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jr8sz" Sep 11 14:28:27.199180 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:28:27.199128 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:57.927173 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:29:57.927091 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:29:57.927724 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:29:57.927553 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:30:01.692918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:01.692868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:01.826050 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:01.826013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:33.952372 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:33.952323 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:34.053423 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:34.053385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:35.234825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:35.234785 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:35.323316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:35.323277 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:55.915883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:55.915843 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:56.018841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:56.018807 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:56.291122 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:56.291082 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:56.389537 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:56.389497 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:57.520669 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:57.520622 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:57.610907 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:57.610864 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:58.084986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:58.084926 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:59.602011 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:30:59.601952 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:00.592581 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.592542 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:00.758278 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.758243 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:31:00.761580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.761557 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.764073 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.764048 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e85961-serving-cert\"" Sep 11 14:31:00.778417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.778385 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:31:00.808780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.808741 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:00.831582 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.831540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.831789 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.831603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc74\" (UniqueName: \"kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.831789 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.831671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.831789 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.831699 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.916959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.916828 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:00.932824 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.932797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc74\" (UniqueName: \"kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.932994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.932866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.932994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.932897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.932994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.932987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.933159 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:31:00.933099 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-serving-cert: secret "maas-api-e2e-inf-a-e85961-serving-cert" not found Sep 11 14:31:00.933233 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:31:00.933182 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls podName:febf6857-219e-4e5b-853d-1a193b72a182 nodeName:}" failed. No retries permitted until 2026-09-11 14:31:01.433160035 +0000 UTC m=+964.004817500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls") pod "maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" (UID: "febf6857-219e-4e5b-853d-1a193b72a182") : secret "maas-api-e2e-inf-a-e85961-serving-cert" not found Sep 11 14:31:00.933660 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.933641 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.935526 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.935504 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:00.947676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:00.947645 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc74\" (UniqueName: \"kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:01.290072 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.290036 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:01.437619 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.437576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:01.440343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.440310 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") pod \"maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:01.673579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.673498 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:01.696077 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.696043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:01.795119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.795084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:01.849189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.849152 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:31:01.852901 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.852879 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:01.856266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.856231 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e85961\"" Sep 11 14:31:01.876577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.876528 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:31:01.902589 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.902556 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:31:01.905925 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.905899 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:01.908086 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.908066 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e85961-dockercfg-lvtlw\"" Sep 11 14:31:01.924619 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.924546 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:31:01.942653 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.942610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r97t\" (UniqueName: \"kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:01.942831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.942790 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:01.942831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.942823 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:01.942926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.942849 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgjr\" (UniqueName: \"kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:01.967225 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:01.967191 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:31:02.023170 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.023132 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" event={"ID":"febf6857-219e-4e5b-853d-1a193b72a182","Type":"ContainerStarted","Data":"8591fb83c0b37c61a29fd7aa65798d06717a8b8f1fcbd88fa5535d09f084e68b"} Sep 11 14:31:02.043735 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.043691 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:02.043735 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.043739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:02.044019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.043765 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgjr\" (UniqueName: \"kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:02.044019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.043811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r97t\" (UniqueName: \"kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:02.044593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.044566 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:02.044689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.044567 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:02.057989 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.057958 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r97t\" (UniqueName: \"kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t\") pod \"payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:02.059743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.059716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgjr\" (UniqueName: \"kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr\") pod \"payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:02.163710 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.163663 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:02.217743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.217611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:02.450706 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.450614 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:31:02.453319 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:02.453291 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b63506a_fb40_4dae_9e71_957c68045261.slice/crio-be83551e2d6efd861b97b102585ec35d41f9f2452fdea60b9304c0c71e43704b WatchSource:0}: Error finding container be83551e2d6efd861b97b102585ec35d41f9f2452fdea60b9304c0c71e43704b: Status 404 returned error can't find the container with id be83551e2d6efd861b97b102585ec35d41f9f2452fdea60b9304c0c71e43704b Sep 11 14:31:02.524666 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:02.524635 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:31:02.526026 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:02.525999 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c66155d_ce03_4769_9727_ed8d1c91b30d.slice/crio-ec7e0aaafd4aa14896a09d6fa931d40fe9319b47545d32989d54fd690374c3fd WatchSource:0}: Error finding container ec7e0aaafd4aa14896a09d6fa931d40fe9319b47545d32989d54fd690374c3fd: Status 404 returned error can't find the container with id ec7e0aaafd4aa14896a09d6fa931d40fe9319b47545d32989d54fd690374c3fd Sep 11 14:31:03.030265 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.030223 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" event={"ID":"4c66155d-ce03-4769-9727-ed8d1c91b30d","Type":"ContainerStarted","Data":"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003"} Sep 11 14:31:03.030767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.030285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" event={"ID":"4c66155d-ce03-4769-9727-ed8d1c91b30d","Type":"ContainerStarted","Data":"ec7e0aaafd4aa14896a09d6fa931d40fe9319b47545d32989d54fd690374c3fd"} Sep 11 14:31:03.030767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.030312 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:03.031993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.031959 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" event={"ID":"febf6857-219e-4e5b-853d-1a193b72a182","Type":"ContainerStarted","Data":"80004216d0b1012c756baeb9fd5daa08dd2d2b3e8f610d90ce79e029db2f55c7"} Sep 11 14:31:03.032133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.032046 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:03.033533 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.033505 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" event={"ID":"4b63506a-fb40-4dae-9e71-957c68045261","Type":"ContainerStarted","Data":"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351"} Sep 11 14:31:03.033680 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.033555 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" event={"ID":"4b63506a-fb40-4dae-9e71-957c68045261","Type":"ContainerStarted","Data":"be83551e2d6efd861b97b102585ec35d41f9f2452fdea60b9304c0c71e43704b"} Sep 11 14:31:03.033680 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.033584 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:03.056586 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.056534 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" podStartSLOduration=2.056518268 podStartE2EDuration="2.056518268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:03.054601491 +0000 UTC m=+965.626258994" watchObservedRunningTime="2026-09-11 14:31:03.056518268 +0000 UTC m=+965.628175737" Sep 11 14:31:03.088134 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.088064 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" podStartSLOduration=2.7226284229999997 podStartE2EDuration="3.088040016s" totalImagesPullingTime="365.411593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:03.085114853 +0000 UTC m=+965.656772357" watchObservedRunningTime="2026-09-11 14:31:03.088040016 +0000 UTC m=+965.659697491" Sep 11 14:31:03.113493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:03.113422 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" podStartSLOduration=2.113400982 podStartE2EDuration="2.113400982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:03.111093768 +0000 UTC m=+965.682751262" watchObservedRunningTime="2026-09-11 14:31:03.113400982 +0000 UTC m=+965.685058463" Sep 11 14:31:04.714500 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:04.714470 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:05.490649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:05.490614 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:05.597410 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:05.597377 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:05.817198 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:05.817160 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:06.227189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:06.227097 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:06.399534 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:06.399497 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:09.042841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:09.042800 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:09.044591 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:09.044567 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:31:09.138098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:09.138056 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:09.450689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:09.450589 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:10.738097 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:10.738064 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:10.832553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:10.832517 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:11.006067 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.005974 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:11.497127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.497094 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:11.601430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.601399 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:11.804463 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.804427 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:11.808334 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.808314 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.811168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.811138 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b6b934c\"" Sep 11 14:31:11.811328 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.811302 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b6b934c-dockercfg-sw8lp\"" Sep 11 14:31:11.823023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.822998 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:11.842475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.842430 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.842683 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.842517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxrv\" (UniqueName: \"kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.943138 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.943102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.943316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.943154 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxrv\" (UniqueName: \"kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.943818 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.943797 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:11.954307 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:11.954276 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxrv\" (UniqueName: \"kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv\") pod \"payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:12.119369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:12.119275 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:12.424841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:12.424816 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:12.427043 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:12.427016 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc75c9_7d9e_45b0_94e3_fde1e5b795ae.slice/crio-1aa59266819e5c5a0d15c23af27693da138355ead948db23b57d1aeb4722eae7 WatchSource:0}: Error finding container 1aa59266819e5c5a0d15c23af27693da138355ead948db23b57d1aeb4722eae7: Status 404 returned error can't find the container with id 1aa59266819e5c5a0d15c23af27693da138355ead948db23b57d1aeb4722eae7 Sep 11 14:31:13.087146 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:13.087101 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" event={"ID":"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae","Type":"ContainerStarted","Data":"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4"} Sep 11 14:31:13.087568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:13.087174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" event={"ID":"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae","Type":"ContainerStarted","Data":"1aa59266819e5c5a0d15c23af27693da138355ead948db23b57d1aeb4722eae7"} Sep 11 14:31:13.087568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:13.087233 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:13.114210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:13.114151 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" podStartSLOduration=2.114128885 podStartE2EDuration="2.114128885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:13.113594522 +0000 UTC m=+975.685252020" watchObservedRunningTime="2026-09-11 14:31:13.114128885 +0000 UTC m=+975.685786360" Sep 11 14:31:14.040716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:14.040687 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:31:14.040929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:14.040756 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:31:14.776026 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:14.775988 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:15.175819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:15.175708 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:15.547852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:15.547816 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:17.196225 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:17.196188 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:17.422143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:17.422104 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:17.522351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:17.522264 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:17.667130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:17.667091 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:19.860987 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:19.860922 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:20.513454 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:20.513413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:20.630444 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:20.630414 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:20.723970 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:20.723918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:20.917539 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:20.917499 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:22.322797 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:22.322757 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:22.528656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:22.528614 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:23.480113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.480074 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:31:23.484132 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.484102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.486245 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.486220 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-84d3a8-serving-cert\"" Sep 11 14:31:23.503999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.503965 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:31:23.548483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.548426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:23.554593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.554557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.554829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.554598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.554829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.554704 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.554829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.554777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmh6\" (UniqueName: \"kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.626593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.626524 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:23.655784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.655737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.655784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.655788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.656089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.655833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.656089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.655909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmh6\" (UniqueName: \"kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.656089 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:31:23.655997 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-serving-cert: secret "maas-api-e2e-autores-84d3a8-serving-cert" not found Sep 11 14:31:23.656089 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:31:23.656075 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls podName:bd57bae1-5447-4d8c-b4e8-7cfdeea74e17 nodeName:}" failed. No retries permitted until 2026-09-11 14:31:24.156051401 +0000 UTC m=+986.727708870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls") pod "maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" (UID: "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17") : secret "maas-api-e2e-autores-84d3a8-serving-cert" not found Sep 11 14:31:23.656578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.656555 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.658502 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.658483 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:23.666561 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:23.666534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmh6\" (UniqueName: \"kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:24.076308 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.076274 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:24.095401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.095374 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:24.161283 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.161247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:24.163818 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.163787 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") pod \"maas-api-e2e-autores-84d3a8-77b6948485-rfmhs\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:24.211500 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.211464 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:24.340398 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.340302 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:24.396472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.396429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:24.449537 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.449459 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:24.711538 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.711448 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:31:24.715930 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:24.715896 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd57bae1_5447_4d8c_b4e8_7cfdeea74e17.slice/crio-6c0061d4474bdc0950ff4a885817b467a0db986987172691bff5748aae6c50c5 WatchSource:0}: Error finding container 6c0061d4474bdc0950ff4a885817b467a0db986987172691bff5748aae6c50c5: Status 404 returned error can't find the container with id 6c0061d4474bdc0950ff4a885817b467a0db986987172691bff5748aae6c50c5 Sep 11 14:31:24.747038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:24.747007 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:25.100662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.100624 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:25.150502 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.150464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" event={"ID":"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17","Type":"ContainerStarted","Data":"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8"} Sep 11 14:31:25.150717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.150526 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:25.150717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.150546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" event={"ID":"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17","Type":"ContainerStarted","Data":"6c0061d4474bdc0950ff4a885817b467a0db986987172691bff5748aae6c50c5"} Sep 11 14:31:25.184106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.184041 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" podStartSLOduration=1.915918122 podStartE2EDuration="2.184019809s" totalImagesPullingTime="268.101687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:25.181660368 +0000 UTC m=+987.753317859" watchObservedRunningTime="2026-09-11 14:31:25.184019809 +0000 UTC m=+987.755677282" Sep 11 14:31:25.386478 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.386442 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:25.490535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.490499 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:25.946970 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:25.946906 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:26.728364 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:26.728329 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:27.823226 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:27.823191 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:28.690005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:28.689961 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:28.785022 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:28.784982 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:29.154712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:29.154678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:30.263047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:30.263003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:30.364668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:30.364618 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:30.561878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:30.561845 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:31.161636 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.161608 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:31:31.471958 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.471860 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:31:31.476274 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.476251 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.478076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.478055 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5a7035-serving-cert\"" Sep 11 14:31:31.497845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.497809 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:31:31.519041 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.519004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:31.611143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.611111 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:31.640872 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.640834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.640872 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.640873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.641096 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.641023 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.641134 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.641104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swp4\" (UniqueName: \"kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.713361 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.713328 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:31.742604 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.742530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.742604 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.742597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.742811 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.742644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swp4\" (UniqueName: \"kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.742811 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.742684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.743233 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.743212 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.745195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.745172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.745195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.745190 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.753546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.753518 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swp4\" (UniqueName: \"kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4\") pod \"maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:31.786580 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:31.786539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:32.068592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.068564 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:31:32.070763 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:32.070731 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2bc1aa_44ee_4694_9d43_535f21c754f6.slice/crio-5e59c5f7d7ccae8cba38702e4268b7e235c72346f5e533040871fa28dcf51cba WatchSource:0}: Error finding container 5e59c5f7d7ccae8cba38702e4268b7e235c72346f5e533040871fa28dcf51cba: Status 404 returned error can't find the container with id 5e59c5f7d7ccae8cba38702e4268b7e235c72346f5e533040871fa28dcf51cba Sep 11 14:31:32.188011 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.187974 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" event={"ID":"ca2bc1aa-44ee-4694-9d43-535f21c754f6","Type":"ContainerStarted","Data":"5e59c5f7d7ccae8cba38702e4268b7e235c72346f5e533040871fa28dcf51cba"} Sep 11 14:31:32.201165 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.201125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:32.311491 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.311452 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:32.453186 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.453103 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:31:32.456595 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.456574 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.458777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.458746 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5a7035-dockercfg-hm5dr\"" Sep 11 14:31:32.458926 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.458813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a7035\"" Sep 11 14:31:32.480063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.480000 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:31:32.551532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.551499 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.551731 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.551708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2rl\" (UniqueName: \"kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.652790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.652758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2rl\" (UniqueName: \"kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.653005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.652807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.653507 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.653487 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.663744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.663717 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2rl\" (UniqueName: \"kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl\") pod \"payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:32.768207 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:32.768111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:33.049300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.049267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:33.064981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.064951 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:31:33.067975 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:33.067929 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd50313_a735_490f_8d50_35764b4799cf.slice/crio-1b5efcd4b0d49117eea34fa461a46cab8c8cc218e13575cae28e34e087203daa WatchSource:0}: Error finding container 1b5efcd4b0d49117eea34fa461a46cab8c8cc218e13575cae28e34e087203daa: Status 404 returned error can't find the container with id 1b5efcd4b0d49117eea34fa461a46cab8c8cc218e13575cae28e34e087203daa Sep 11 14:31:33.197146 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.197112 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" event={"ID":"1cd50313-a735-490f-8d50-35764b4799cf","Type":"ContainerStarted","Data":"a1655b5ed84e122b0eaf16048ae09b9c2a483df6382ab0aff091b7e5c2d80106"} Sep 11 14:31:33.197353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.197172 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" event={"ID":"1cd50313-a735-490f-8d50-35764b4799cf","Type":"ContainerStarted","Data":"1b5efcd4b0d49117eea34fa461a46cab8c8cc218e13575cae28e34e087203daa"} Sep 11 14:31:33.197353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.197305 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:33.198827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.198801 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" event={"ID":"ca2bc1aa-44ee-4694-9d43-535f21c754f6","Type":"ContainerStarted","Data":"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c"} Sep 11 14:31:33.198993 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.198861 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:33.228034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.227974 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" podStartSLOduration=1.227954173 podStartE2EDuration="1.227954173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:33.227953285 +0000 UTC m=+995.799610779" watchObservedRunningTime="2026-09-11 14:31:33.227954173 +0000 UTC m=+995.799611652" Sep 11 14:31:33.263520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:33.263460 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" podStartSLOduration=1.864397826 podStartE2EDuration="2.263442397s" totalImagesPullingTime="399.044571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:33.262329226 +0000 UTC m=+995.833986715" watchObservedRunningTime="2026-09-11 14:31:33.263442397 +0000 UTC m=+995.835099870" Sep 11 14:31:34.443717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:34.443678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:35.641105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:35.641069 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:36.876432 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:36.876376 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.031397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.031361 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.155904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.155872 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:38.156181 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.156147 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" podUID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" containerName="payload-processing" containerID="cri-o://39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4" gracePeriod=30 Sep 11 14:31:38.255300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.255248 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.377381 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.377313 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.412573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.412543 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:38.511878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.511841 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vxrv\" (UniqueName: \"kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv\") pod \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " Sep 11 14:31:38.512086 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.511917 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume\") pod \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\" (UID: \"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae\") " Sep 11 14:31:38.512330 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.512310 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" (UID: "ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:38.514316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.514288 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv" (OuterVolumeSpecName: "kube-api-access-2vxrv") pod "ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" (UID: "ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae"). InnerVolumeSpecName "kube-api-access-2vxrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:38.515053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.515024 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.612883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.612796 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxrv\" (UniqueName: \"kubernetes.io/projected/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-kube-api-access-2vxrv\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.612883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.612828 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.623890 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.623850 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.753108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.753070 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:38.854914 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:38.854880 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.210589 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.210562 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:31:39.234048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.234016 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" containerID="39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4" exitCode=0 Sep 11 14:31:39.234048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.234053 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" event={"ID":"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae","Type":"ContainerDied","Data":"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4"} Sep 11 14:31:39.234272 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.234096 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" event={"ID":"ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae","Type":"ContainerDied","Data":"1aa59266819e5c5a0d15c23af27693da138355ead948db23b57d1aeb4722eae7"} Sep 11 14:31:39.234272 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.234117 2571 scope.go:117] "RemoveContainer" containerID="39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4" Sep 11 14:31:39.234272 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.234055 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp" Sep 11 14:31:39.244496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.244477 2571 scope.go:117] "RemoveContainer" containerID="39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4" Sep 11 14:31:39.244817 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:31:39.244787 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4\": container with ID starting with 39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4 not found: ID does not exist" containerID="39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4" Sep 11 14:31:39.244892 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.244825 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4"} err="failed to get container status \"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4\": rpc error: code = NotFound desc = could not find container \"39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4\": container with ID starting with 39ba60f7f6daa03bdffb12376339b40a55a62eec4d77924614c8459764866fb4 not found: ID does not exist" Sep 11 14:31:39.304286 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.304252 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:39.314061 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.314030 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b6b934c-54b788b759-7rrqp"] Sep 11 14:31:39.372862 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.372819 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.456397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.456353 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.567210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.567175 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.755841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.755802 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.840483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.840388 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:39.959305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:39.959270 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" path="/var/lib/kubelet/pods/ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae/volumes" Sep 11 14:31:40.056619 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:40.056582 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:40.214613 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:40.214525 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:40.493296 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:40.493203 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:40.582022 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:40.581985 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:40.732105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:40.732004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:42.341546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:42.341511 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:43.465360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:43.465321 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:43.668748 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:43.668719 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:43.760012 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:43.759900 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:44.208100 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:44.208069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:31:45.102479 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:45.102438 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:46.271696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:46.271659 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:46.369008 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:46.368964 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:47.114859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.114827 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:47.322523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.322483 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:47.562769 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.562730 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:31:47.563208 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.563193 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" containerName="payload-processing" Sep 11 14:31:47.563264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.563210 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" containerName="payload-processing" Sep 11 14:31:47.563301 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.563285 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbc75c9-7d9e-45b0-94e3-fde1e5b795ae" containerName="payload-processing" Sep 11 14:31:47.567572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.567548 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.569684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.569665 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a6adb955\"" Sep 11 14:31:47.569803 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.569785 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a6adb955-dockercfg-4f872\"" Sep 11 14:31:47.602517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.602477 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.602703 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.602566 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgqn\" (UniqueName: \"kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.629214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.629182 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:31:47.703907 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.703872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.704091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.703979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgqn\" (UniqueName: \"kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.704686 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.704665 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.720365 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.720326 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgqn\" (UniqueName: \"kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn\") pod \"payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.722773 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.722741 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:31:47.726547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.726520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.742362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.742327 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:31:47.805417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.805375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.805417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.805422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssnn\" (UniqueName: \"kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.878115 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.878004 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:47.906963 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.906898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.907160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.906983 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssnn\" (UniqueName: \"kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.907832 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.907807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:47.918806 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:47.918773 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssnn\" (UniqueName: \"kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn\") pod \"payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:48.038402 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.038363 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:48.164724 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.164696 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:31:48.166949 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:48.166898 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4f8600_5558_4e8d_87d6_3bf792fe724b.slice/crio-8dc22584be7825916db35182dea5fb5fac058894d17b4eb065fcb0723bfc55db WatchSource:0}: Error finding container 8dc22584be7825916db35182dea5fb5fac058894d17b4eb065fcb0723bfc55db: Status 404 returned error can't find the container with id 8dc22584be7825916db35182dea5fb5fac058894d17b4eb065fcb0723bfc55db Sep 11 14:31:48.182688 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.182654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:48.285390 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.285342 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" event={"ID":"1e4f8600-5558-4e8d-87d6-3bf792fe724b","Type":"ContainerStarted","Data":"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9"} Sep 11 14:31:48.285578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.285415 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:31:48.285578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.285439 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" event={"ID":"1e4f8600-5558-4e8d-87d6-3bf792fe724b","Type":"ContainerStarted","Data":"8dc22584be7825916db35182dea5fb5fac058894d17b4eb065fcb0723bfc55db"} Sep 11 14:31:48.311061 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.311008 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" podStartSLOduration=1.310990553 podStartE2EDuration="1.310990553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:48.309195995 +0000 UTC m=+1010.880853495" watchObservedRunningTime="2026-09-11 14:31:48.310990553 +0000 UTC m=+1010.882648143" Sep 11 14:31:48.359035 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.358994 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:31:48.368582 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:31:48.368550 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303dfb2f_0d61_47c9_b614_8ea685e8181f.slice/crio-5c362b2928a676c7ed7a99b93d34f3f14f400edd964720021e5b4405193e6237 WatchSource:0}: Error finding container 5c362b2928a676c7ed7a99b93d34f3f14f400edd964720021e5b4405193e6237: Status 404 returned error can't find the container with id 5c362b2928a676c7ed7a99b93d34f3f14f400edd964720021e5b4405193e6237 Sep 11 14:31:48.829837 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:48.829792 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:49.293282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.293234 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" event={"ID":"303dfb2f-0d61-47c9-b614-8ea685e8181f","Type":"ContainerStarted","Data":"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0"} Sep 11 14:31:49.293475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.293295 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" event={"ID":"303dfb2f-0d61-47c9-b614-8ea685e8181f","Type":"ContainerStarted","Data":"5c362b2928a676c7ed7a99b93d34f3f14f400edd964720021e5b4405193e6237"} Sep 11 14:31:49.293475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.293458 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:31:49.329302 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.329247 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" podStartSLOduration=2.329226051 podStartE2EDuration="2.329226051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:49.324332547 +0000 UTC m=+1011.895990040" watchObservedRunningTime="2026-09-11 14:31:49.329226051 +0000 UTC m=+1011.900883524" Sep 11 14:31:49.576092 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.576003 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:49.960289 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:49.960193 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:50.063928 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:50.063894 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:50.783425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:50.783386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:51.925233 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:51.925195 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:53.003741 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:53.003707 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:54.285221 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:54.285173 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:55.405252 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:55.405216 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:56.694315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:56.694280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:57.905162 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:57.905111 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:59.134881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:59.134836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:59.296355 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:31:59.296326 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:32:00.294723 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:00.294687 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:00.308071 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:00.308042 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:32:01.309745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:01.309702 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:02.316950 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:02.316895 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:03.494683 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:03.494647 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:05.732763 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:05.732726 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:08.121417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:08.121383 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:09.383924 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.383885 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:09.509006 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.508969 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:32:09.509250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.509222 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" podUID="303dfb2f-0d61-47c9-b614-8ea685e8181f" containerName="payload-processing" containerID="cri-o://8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0" gracePeriod=30 Sep 11 14:32:09.530059 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.530032 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:32:09.530291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.530267 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" podUID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" containerName="payload-pre-processing" containerID="cri-o://24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9" gracePeriod=30 Sep 11 14:32:09.575628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.575594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:09.685534 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.685463 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:09.823869 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.823828 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:32:09.827438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.827147 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:32:09.839051 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.838994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:09.929007 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.928910 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkgqn\" (UniqueName: \"kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn\") pod \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " Sep 11 14:32:09.929192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.929035 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gssnn\" (UniqueName: \"kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn\") pod \"303dfb2f-0d61-47c9-b614-8ea685e8181f\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " Sep 11 14:32:09.929192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.929092 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume\") pod \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\" (UID: \"1e4f8600-5558-4e8d-87d6-3bf792fe724b\") " Sep 11 14:32:09.929192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.929117 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume\") pod \"303dfb2f-0d61-47c9-b614-8ea685e8181f\" (UID: \"303dfb2f-0d61-47c9-b614-8ea685e8181f\") " Sep 11 14:32:09.929490 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.929454 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e4f8600-5558-4e8d-87d6-3bf792fe724b" (UID: "1e4f8600-5558-4e8d-87d6-3bf792fe724b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:09.929599 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.929529 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "303dfb2f-0d61-47c9-b614-8ea685e8181f" (UID: "303dfb2f-0d61-47c9-b614-8ea685e8181f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:09.931923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.931791 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn" (OuterVolumeSpecName: "kube-api-access-gssnn") pod "303dfb2f-0d61-47c9-b614-8ea685e8181f" (UID: "303dfb2f-0d61-47c9-b614-8ea685e8181f"). InnerVolumeSpecName "kube-api-access-gssnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:09.931923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.931885 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn" (OuterVolumeSpecName: "kube-api-access-rkgqn") pod "1e4f8600-5558-4e8d-87d6-3bf792fe724b" (UID: "1e4f8600-5558-4e8d-87d6-3bf792fe724b"). InnerVolumeSpecName "kube-api-access-rkgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:09.949543 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:09.949511 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:10.030428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.030390 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgqn\" (UniqueName: \"kubernetes.io/projected/1e4f8600-5558-4e8d-87d6-3bf792fe724b-kube-api-access-rkgqn\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:10.030428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.030421 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gssnn\" (UniqueName: \"kubernetes.io/projected/303dfb2f-0d61-47c9-b614-8ea685e8181f-kube-api-access-gssnn\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:10.030428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.030431 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4f8600-5558-4e8d-87d6-3bf792fe724b-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:10.030675 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.030440 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303dfb2f-0d61-47c9-b614-8ea685e8181f-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:10.104696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.104659 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:10.414396 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.414360 2571 generic.go:356] "Generic (PLEG): container finished" podID="303dfb2f-0d61-47c9-b614-8ea685e8181f" containerID="8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0" exitCode=0 Sep 11 14:32:10.414873 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.414401 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" Sep 11 14:32:10.414873 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.414453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" event={"ID":"303dfb2f-0d61-47c9-b614-8ea685e8181f","Type":"ContainerDied","Data":"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0"} Sep 11 14:32:10.414873 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.414507 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh" event={"ID":"303dfb2f-0d61-47c9-b614-8ea685e8181f","Type":"ContainerDied","Data":"5c362b2928a676c7ed7a99b93d34f3f14f400edd964720021e5b4405193e6237"} Sep 11 14:32:10.414873 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.414538 2571 scope.go:117] "RemoveContainer" containerID="8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0" Sep 11 14:32:10.415918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.415890 2571 generic.go:356] "Generic (PLEG): container finished" podID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" containerID="24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9" exitCode=0 Sep 11 14:32:10.416032 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.415950 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" Sep 11 14:32:10.416090 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.415953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" event={"ID":"1e4f8600-5558-4e8d-87d6-3bf792fe724b","Type":"ContainerDied","Data":"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9"} Sep 11 14:32:10.416090 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.416073 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch" event={"ID":"1e4f8600-5558-4e8d-87d6-3bf792fe724b","Type":"ContainerDied","Data":"8dc22584be7825916db35182dea5fb5fac058894d17b4eb065fcb0723bfc55db"} Sep 11 14:32:10.423614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.423592 2571 scope.go:117] "RemoveContainer" containerID="8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0" Sep 11 14:32:10.423991 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:32:10.423968 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0\": container with ID starting with 8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0 not found: ID does not exist" containerID="8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0" Sep 11 14:32:10.424056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.424002 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0"} err="failed to get container status \"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0\": rpc error: code = NotFound desc = could not find container \"8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0\": container with ID starting with 8860e0312c68d9075bd213a26471f0a0d886ace7783e1ffe0178505f6c3615d0 not found: ID does not exist" Sep 11 14:32:10.424056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.424029 2571 scope.go:117] "RemoveContainer" containerID="24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9" Sep 11 14:32:10.432504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.432483 2571 scope.go:117] "RemoveContainer" containerID="24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9" Sep 11 14:32:10.432817 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:32:10.432799 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9\": container with ID starting with 24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9 not found: ID does not exist" containerID="24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9" Sep 11 14:32:10.432866 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.432826 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9"} err="failed to get container status \"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9\": rpc error: code = NotFound desc = could not find container \"24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9\": container with ID starting with 24f9a74bd18d31c3a9f35ac99f88951c78126006ddbe529887c5404f348c45f9 not found: ID does not exist" Sep 11 14:32:10.442046 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.442017 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:32:10.474598 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.474566 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a6adb955-5dcb44cb78-4tczh"] Sep 11 14:32:10.501201 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.501160 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:32:10.509206 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.509175 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a6adb955-57b8db9774-cgqch"] Sep 11 14:32:10.578382 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.578344 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:10.672990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:10.672883 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:11.286001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:11.285972 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:11.857837 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:11.857799 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:11.959601 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:11.959572 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" path="/var/lib/kubelet/pods/1e4f8600-5558-4e8d-87d6-3bf792fe724b/volumes" Sep 11 14:32:11.959975 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:11.959962 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303dfb2f-0d61-47c9-b614-8ea685e8181f" path="/var/lib/kubelet/pods/303dfb2f-0d61-47c9-b614-8ea685e8181f/volumes" Sep 11 14:32:12.406279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:12.406237 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:13.859609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:13.859575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:14.482664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:14.482627 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:14.600115 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:14.600076 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:15.150054 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:15.150018 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:16.892836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:16.892799 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:16.985704 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:16.985664 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:17.096098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:17.096063 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:18.171973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.171868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:18.318405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.318357 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:18.420095 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420061 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:18.420503 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420490 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303dfb2f-0d61-47c9-b614-8ea685e8181f" containerName="payload-processing" Sep 11 14:32:18.420549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420506 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="303dfb2f-0d61-47c9-b614-8ea685e8181f" containerName="payload-processing" Sep 11 14:32:18.420549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420527 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" containerName="payload-pre-processing" Sep 11 14:32:18.420549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420533 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" containerName="payload-pre-processing" Sep 11 14:32:18.420640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420593 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="303dfb2f-0d61-47c9-b614-8ea685e8181f" containerName="payload-processing" Sep 11 14:32:18.420640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.420605 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4f8600-5558-4e8d-87d6-3bf792fe724b" containerName="payload-pre-processing" Sep 11 14:32:18.425140 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.425088 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.428091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.428063 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-61c24488\"" Sep 11 14:32:18.431683 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.431662 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-61c24488-dockercfg-lvc4x\"" Sep 11 14:32:18.445676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.445649 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:18.466216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.466185 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:18.469845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.469823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.491267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.491237 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:18.508367 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.508334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw8m\" (UniqueName: \"kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.508367 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.508374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sm7\" (UniqueName: \"kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.508599 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.508476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.508599 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.508503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.609878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.609840 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw8m\" (UniqueName: \"kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.610133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.609886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sm7\" (UniqueName: \"kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.610133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.609995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.610133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.610017 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.610746 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.610725 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.610804 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.610727 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.623547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.623519 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sm7\" (UniqueName: \"kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7\") pod \"payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.624496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.624469 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw8m\" (UniqueName: \"kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m\") pod \"payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:18.736256 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.736154 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:18.782442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:18.782407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:19.075033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.075000 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:19.077750 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:32:19.077722 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a139f0e_86b4_4e92_87c7_0d38e3b90b4a.slice/crio-ac48fb39729bfed5761e37e87b457f6ff7d87fdebcdbc9dc4e87b9411adf7477 WatchSource:0}: Error finding container ac48fb39729bfed5761e37e87b457f6ff7d87fdebcdbc9dc4e87b9411adf7477: Status 404 returned error can't find the container with id ac48fb39729bfed5761e37e87b457f6ff7d87fdebcdbc9dc4e87b9411adf7477 Sep 11 14:32:19.116734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.116706 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:19.118546 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:32:19.118520 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9d0795_b13a_4086_b995_2056f77b485b.slice/crio-f0cfff3298f5415c172b21e89d9d8906836eb8f7da9b62a5cf80786d0fa6913d WatchSource:0}: Error finding container f0cfff3298f5415c172b21e89d9d8906836eb8f7da9b62a5cf80786d0fa6913d: Status 404 returned error can't find the container with id f0cfff3298f5415c172b21e89d9d8906836eb8f7da9b62a5cf80786d0fa6913d Sep 11 14:32:19.128984 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.128954 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:19.466276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.466169 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" event={"ID":"2a9d0795-b13a-4086-b995-2056f77b485b","Type":"ContainerStarted","Data":"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98"} Sep 11 14:32:19.466276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.466242 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:19.466276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.466270 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" event={"ID":"2a9d0795-b13a-4086-b995-2056f77b485b","Type":"ContainerStarted","Data":"f0cfff3298f5415c172b21e89d9d8906836eb8f7da9b62a5cf80786d0fa6913d"} Sep 11 14:32:19.467992 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.467956 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" event={"ID":"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a","Type":"ContainerStarted","Data":"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75"} Sep 11 14:32:19.468120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.468012 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:19.468120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.468029 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" event={"ID":"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a","Type":"ContainerStarted","Data":"ac48fb39729bfed5761e37e87b457f6ff7d87fdebcdbc9dc4e87b9411adf7477"} Sep 11 14:32:19.520109 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.520056 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" podStartSLOduration=1.520038201 podStartE2EDuration="1.520038201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:19.518230256 +0000 UTC m=+1042.089887747" watchObservedRunningTime="2026-09-11 14:32:19.520038201 +0000 UTC m=+1042.091695674" Sep 11 14:32:19.543819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:19.543755 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" podStartSLOduration=1.543736353 podStartE2EDuration="1.543736353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:19.543198809 +0000 UTC m=+1042.114856302" watchObservedRunningTime="2026-09-11 14:32:19.543736353 +0000 UTC m=+1042.115393819" Sep 11 14:32:20.405371 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:20.405323 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:21.604485 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:21.604447 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:30.475579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:30.475547 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:30.476000 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:30.475617 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:39.248407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:39.248363 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:40.986472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:40.986440 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:42.189532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:42.189493 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:43.156995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:43.156959 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:44.294693 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:44.294646 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:45.433525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:45.433480 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:46.499186 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:46.499141 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:47.516012 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:47.515975 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:48.554806 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:48.554754 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:49.597108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:49.597071 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:49.708409 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:49.708368 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:50.630041 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:50.629994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:54.345253 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:54.345210 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:56.432500 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:56.432450 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.420636 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.420596 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.547548 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.547503 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:57.548034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.547766 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" podUID="2a9d0795-b13a-4086-b995-2056f77b485b" containerName="payload-processing" containerID="cri-o://733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98" gracePeriod=30 Sep 11 14:32:57.568778 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.568750 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:57.569000 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.568974 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" podUID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" containerName="payload-pre-processing" containerID="cri-o://cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75" gracePeriod=30 Sep 11 14:32:57.584595 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.584560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.679679 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.679626 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.802464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.802417 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.850036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.849968 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:57.853159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.853128 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:57.897878 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.897835 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.898605 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.898577 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume\") pod \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " Sep 11 14:32:57.898605 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.898607 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume\") pod \"2a9d0795-b13a-4086-b995-2056f77b485b\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " Sep 11 14:32:57.898810 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.898649 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58sm7\" (UniqueName: \"kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7\") pod \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\" (UID: \"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a\") " Sep 11 14:32:57.898810 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.898760 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqw8m\" (UniqueName: \"kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m\") pod \"2a9d0795-b13a-4086-b995-2056f77b485b\" (UID: \"2a9d0795-b13a-4086-b995-2056f77b485b\") " Sep 11 14:32:57.899156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.899129 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" (UID: "7a139f0e-86b4-4e92-87c7-0d38e3b90b4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:57.899369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.899334 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:57.899832 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.899804 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a9d0795-b13a-4086-b995-2056f77b485b" (UID: "2a9d0795-b13a-4086-b995-2056f77b485b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:57.901810 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.901781 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7" (OuterVolumeSpecName: "kube-api-access-58sm7") pod "7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" (UID: "7a139f0e-86b4-4e92-87c7-0d38e3b90b4a"). InnerVolumeSpecName "kube-api-access-58sm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:57.902046 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:57.902020 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m" (OuterVolumeSpecName: "kube-api-access-cqw8m") pod "2a9d0795-b13a-4086-b995-2056f77b485b" (UID: "2a9d0795-b13a-4086-b995-2056f77b485b"). InnerVolumeSpecName "kube-api-access-cqw8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:58.000352 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.000318 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9d0795-b13a-4086-b995-2056f77b485b-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:58.000352 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.000347 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sm7\" (UniqueName: \"kubernetes.io/projected/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a-kube-api-access-58sm7\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:58.000352 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.000357 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw8m\" (UniqueName: \"kubernetes.io/projected/2a9d0795-b13a-4086-b995-2056f77b485b-kube-api-access-cqw8m\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:32:58.381514 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.381474 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:58.667493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.667405 2571 generic.go:356] "Generic (PLEG): container finished" podID="2a9d0795-b13a-4086-b995-2056f77b485b" containerID="733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98" exitCode=0 Sep 11 14:32:58.667493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.667468 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" Sep 11 14:32:58.667493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.667486 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" event={"ID":"2a9d0795-b13a-4086-b995-2056f77b485b","Type":"ContainerDied","Data":"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98"} Sep 11 14:32:58.668046 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.667539 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6" event={"ID":"2a9d0795-b13a-4086-b995-2056f77b485b","Type":"ContainerDied","Data":"f0cfff3298f5415c172b21e89d9d8906836eb8f7da9b62a5cf80786d0fa6913d"} Sep 11 14:32:58.668046 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.667570 2571 scope.go:117] "RemoveContainer" containerID="733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98" Sep 11 14:32:58.668985 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.668960 2571 generic.go:356] "Generic (PLEG): container finished" podID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" containerID="cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75" exitCode=0 Sep 11 14:32:58.669084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.669057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" event={"ID":"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a","Type":"ContainerDied","Data":"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75"} Sep 11 14:32:58.669084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.669067 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" Sep 11 14:32:58.669193 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.669098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd" event={"ID":"7a139f0e-86b4-4e92-87c7-0d38e3b90b4a","Type":"ContainerDied","Data":"ac48fb39729bfed5761e37e87b457f6ff7d87fdebcdbc9dc4e87b9411adf7477"} Sep 11 14:32:58.677083 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.677058 2571 scope.go:117] "RemoveContainer" containerID="733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98" Sep 11 14:32:58.677454 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:32:58.677427 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98\": container with ID starting with 733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98 not found: ID does not exist" containerID="733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98" Sep 11 14:32:58.677582 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.677458 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98"} err="failed to get container status \"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98\": rpc error: code = NotFound desc = could not find container \"733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98\": container with ID starting with 733e0dba1f0e94e6ba0b546d7bfbb08a921c1900fa0d9e96383f4dd53bd0ac98 not found: ID does not exist" Sep 11 14:32:58.677582 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.677477 2571 scope.go:117] "RemoveContainer" containerID="cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75" Sep 11 14:32:58.686300 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.686280 2571 scope.go:117] "RemoveContainer" containerID="cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75" Sep 11 14:32:58.686587 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:32:58.686567 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75\": container with ID starting with cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75 not found: ID does not exist" containerID="cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75" Sep 11 14:32:58.686662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.686593 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75"} err="failed to get container status \"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75\": rpc error: code = NotFound desc = could not find container \"cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75\": container with ID starting with cc0aae7c5384599c2bd1b755b4d198cfd4f367cf3dc85a3f2b99c47f5501ba75 not found: ID does not exist" Sep 11 14:32:58.706106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.706072 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:58.719541 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.719506 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61c24488-6c4dd9f647-8bzz6"] Sep 11 14:32:58.743155 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.743121 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:58.753425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:58.753391 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61c24488-5c96674558-wvckd"] Sep 11 14:32:59.249033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:59.249001 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:59.638472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:59.638432 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:59.740411 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:59.740378 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:59.961200 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:59.961116 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9d0795-b13a-4086-b995-2056f77b485b" path="/var/lib/kubelet/pods/2a9d0795-b13a-4086-b995-2056f77b485b/volumes" Sep 11 14:32:59.962133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:32:59.962107 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" path="/var/lib/kubelet/pods/7a139f0e-86b4-4e92-87c7-0d38e3b90b4a/volumes" Sep 11 14:33:00.172877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:00.172833 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:00.415318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:00.415282 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:01.143381 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:01.143343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:02.311923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:02.311883 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:02.702692 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:02.702575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:02.825366 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:02.825318 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:04.570047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:04.570005 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:04.663224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:04.663193 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:05.086417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.086386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:05.524303 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.524208 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:05.629705 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.629663 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:05.771444 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771409 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:05.771829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771817 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" containerName="payload-pre-processing" Sep 11 14:33:05.771923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771831 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" containerName="payload-pre-processing" Sep 11 14:33:05.771923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771842 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9d0795-b13a-4086-b995-2056f77b485b" containerName="payload-processing" Sep 11 14:33:05.771923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771848 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9d0795-b13a-4086-b995-2056f77b485b" containerName="payload-processing" Sep 11 14:33:05.771923 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771918 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a139f0e-86b4-4e92-87c7-0d38e3b90b4a" containerName="payload-pre-processing" Sep 11 14:33:05.772073 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.771929 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9d0795-b13a-4086-b995-2056f77b485b" containerName="payload-processing" Sep 11 14:33:05.776310 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.776292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.789233 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.789201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-93c38c4c-dockercfg-xlmzt\"" Sep 11 14:33:05.789617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.789597 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-93c38c4c\"" Sep 11 14:33:05.817283 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.817247 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:05.820917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.820888 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:05.821107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.821030 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.835711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.835679 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:05.881278 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.881241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lkp\" (UniqueName: \"kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.881468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.881291 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.881468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.881369 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jdb\" (UniqueName: \"kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.881468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.881424 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.982772 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.982735 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lkp\" (UniqueName: \"kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.982990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.982785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.982990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.982825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jdb\" (UniqueName: \"kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.982990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.982862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.983547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.983526 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:05.983628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.983609 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.995786 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.995755 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lkp\" (UniqueName: \"kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp\") pod \"payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:05.995786 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:05.995780 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jdb\" (UniqueName: \"kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb\") pod \"payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:06.086692 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.086584 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:06.131415 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.131381 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:06.378573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.378478 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:06.380880 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:33:06.380851 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff3b884_f7ea_4f3a_b70d_7173c3493782.slice/crio-f442755de1def9f46ac76dfc3edfb7870c2ae66f0afbee0dcfe4d34b4f74dbd1 WatchSource:0}: Error finding container f442755de1def9f46ac76dfc3edfb7870c2ae66f0afbee0dcfe4d34b4f74dbd1: Status 404 returned error can't find the container with id f442755de1def9f46ac76dfc3edfb7870c2ae66f0afbee0dcfe4d34b4f74dbd1 Sep 11 14:33:06.419698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.419666 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:06.430308 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.430115 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:06.433010 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:33:06.432984 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9882bc3_1bf9_4b0a_b972_f6490ca5c02f.slice/crio-d9027bf9024dc6f0c7de56b1595d1ea8887217dc4543cb2d674596cadbc9fd89 WatchSource:0}: Error finding container d9027bf9024dc6f0c7de56b1595d1ea8887217dc4543cb2d674596cadbc9fd89: Status 404 returned error can't find the container with id d9027bf9024dc6f0c7de56b1595d1ea8887217dc4543cb2d674596cadbc9fd89 Sep 11 14:33:06.711657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.711541 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" event={"ID":"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f","Type":"ContainerStarted","Data":"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d"} Sep 11 14:33:06.711657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.711610 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:06.711657 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.711635 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" event={"ID":"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f","Type":"ContainerStarted","Data":"d9027bf9024dc6f0c7de56b1595d1ea8887217dc4543cb2d674596cadbc9fd89"} Sep 11 14:33:06.713328 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.713300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" event={"ID":"8ff3b884-f7ea-4f3a-b70d-7173c3493782","Type":"ContainerStarted","Data":"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125"} Sep 11 14:33:06.713442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.713350 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:06.713442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.713369 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" event={"ID":"8ff3b884-f7ea-4f3a-b70d-7173c3493782","Type":"ContainerStarted","Data":"f442755de1def9f46ac76dfc3edfb7870c2ae66f0afbee0dcfe4d34b4f74dbd1"} Sep 11 14:33:06.772753 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.772697 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" podStartSLOduration=1.772680751 podStartE2EDuration="1.772680751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:06.771353322 +0000 UTC m=+1089.343010823" watchObservedRunningTime="2026-09-11 14:33:06.772680751 +0000 UTC m=+1089.344338218" Sep 11 14:33:06.796592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:06.796523 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" podStartSLOduration=1.7965023900000001 podStartE2EDuration="1.79650239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:06.793565151 +0000 UTC m=+1089.365222653" watchObservedRunningTime="2026-09-11 14:33:06.79650239 +0000 UTC m=+1089.368159866" Sep 11 14:33:07.520449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:07.520410 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:08.829519 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:08.829479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:10.020516 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:10.020476 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:17.720803 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:17.720771 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:17.721279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:17.720847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:23.685724 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:23.685690 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:25.553145 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:25.553101 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:26.315062 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:26.315013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:27.170593 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:27.170556 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:27.600465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:27.600426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:28.675816 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:28.675779 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:28.805730 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:28.805682 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:30.295094 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:30.295061 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:31.655811 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:31.655756 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:33.206124 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:33.206089 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:35.039591 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:35.039551 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:36.541210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:36.541177 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:38.091645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:38.091611 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:39.517037 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:39.517001 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:40.796951 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:40.796908 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:33:40.797386 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:40.797152 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" podUID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" containerName="maas-api" containerID="cri-o://7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8" gracePeriod=30 Sep 11 14:33:40.850161 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:33:40.850124 2571 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd57bae1_5447_4d8c_b4e8_7cfdeea74e17.slice/crio-7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8.scope\": RecentStats: unable to find data in memory cache]" Sep 11 14:33:40.894884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:40.894844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.077048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.077012 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:33:41.078725 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.078696 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.138704 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.138673 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") pod \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " Sep 11 14:33:41.138905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.138733 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmmh6\" (UniqueName: \"kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6\") pod \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " Sep 11 14:33:41.138905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.138754 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs\") pod \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " Sep 11 14:33:41.138905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.138773 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca\") pod \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\" (UID: \"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17\") " Sep 11 14:33:41.139299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.139265 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" (UID: "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:41.141390 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.141361 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6" (OuterVolumeSpecName: "kube-api-access-tmmh6") pod "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" (UID: "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17"). InnerVolumeSpecName "kube-api-access-tmmh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:41.141494 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.141462 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" (UID: "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:33:41.141674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.141655 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" (UID: "bd57bae1-5447-4d8c-b4e8-7cfdeea74e17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:33:41.222473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.222416 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.240744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.240695 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.240744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.240734 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmh6\" (UniqueName: \"kubernetes.io/projected/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-kube-api-access-tmmh6\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.240744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.240751 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.241062 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.240762 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.397682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.397574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.569383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.569348 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.864883 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.864848 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:41.897903 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.897869 2571 generic.go:356] "Generic (PLEG): container finished" podID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" containerID="7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8" exitCode=0 Sep 11 14:33:41.898091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.897921 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" Sep 11 14:33:41.898091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.897973 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" event={"ID":"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17","Type":"ContainerDied","Data":"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8"} Sep 11 14:33:41.898091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.898032 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs" event={"ID":"bd57bae1-5447-4d8c-b4e8-7cfdeea74e17","Type":"ContainerDied","Data":"6c0061d4474bdc0950ff4a885817b467a0db986987172691bff5748aae6c50c5"} Sep 11 14:33:41.898091 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.898061 2571 scope.go:117] "RemoveContainer" containerID="7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8" Sep 11 14:33:41.908316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.908300 2571 scope.go:117] "RemoveContainer" containerID="7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8" Sep 11 14:33:41.908635 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:33:41.908617 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8\": container with ID starting with 7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8 not found: ID does not exist" containerID="7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8" Sep 11 14:33:41.908674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.908648 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8"} err="failed to get container status \"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8\": rpc error: code = NotFound desc = could not find container \"7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8\": container with ID starting with 7035fd0c55976ee41c7200121704e5426ccf92cb9a2f3d48053722a3dc98c2c8 not found: ID does not exist" Sep 11 14:33:41.934309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.934245 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:33:41.936319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.936291 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-84d3a8-77b6948485-rfmhs"] Sep 11 14:33:41.958690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:41.958661 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" path="/var/lib/kubelet/pods/bd57bae1-5447-4d8c-b4e8-7cfdeea74e17/volumes" Sep 11 14:33:42.012397 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:42.012366 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:43.286461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:43.286424 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:44.756107 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:44.756038 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:46.057702 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:46.057658 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.317480 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.317438 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.423323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.422920 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:47.423606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.423574 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerName="payload-processing" containerID="cri-o://12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d" gracePeriod=30 Sep 11 14:33:47.476100 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.476064 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:47.476379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.476349 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerName="payload-pre-processing" containerID="cri-o://72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125" gracePeriod=30 Sep 11 14:33:47.520181 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.519984 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.636356 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.634897 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.719158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.719120 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.45:9004: connect: connection refused" Sep 11 14:33:47.844128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.844099 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:47.857469 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.857438 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:47.862876 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.862825 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.913664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.913630 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6lkp\" (UniqueName: \"kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp\") pod \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " Sep 11 14:33:47.913856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.913706 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume\") pod \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\" (UID: \"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f\") " Sep 11 14:33:47.914361 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.914327 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" (UID: "e9882bc3-1bf9-4b0a-b972-f6490ca5c02f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:47.917104 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.917080 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp" (OuterVolumeSpecName: "kube-api-access-r6lkp") pod "e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" (UID: "e9882bc3-1bf9-4b0a-b972-f6490ca5c02f"). InnerVolumeSpecName "kube-api-access-r6lkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:47.936656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.935774 2571 generic.go:356] "Generic (PLEG): container finished" podID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerID="12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d" exitCode=0 Sep 11 14:33:47.936656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.935928 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" event={"ID":"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f","Type":"ContainerDied","Data":"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d"} Sep 11 14:33:47.936656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.936014 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" event={"ID":"e9882bc3-1bf9-4b0a-b972-f6490ca5c02f","Type":"ContainerDied","Data":"d9027bf9024dc6f0c7de56b1595d1ea8887217dc4543cb2d674596cadbc9fd89"} Sep 11 14:33:47.936656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.936048 2571 scope.go:117] "RemoveContainer" containerID="12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d" Sep 11 14:33:47.936656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.936088 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" Sep 11 14:33:47.938648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.938518 2571 generic.go:356] "Generic (PLEG): container finished" podID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerID="72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125" exitCode=0 Sep 11 14:33:47.938648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.938558 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" event={"ID":"8ff3b884-f7ea-4f3a-b70d-7173c3493782","Type":"ContainerDied","Data":"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125"} Sep 11 14:33:47.938648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.938592 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" Sep 11 14:33:47.938648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.938601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v" event={"ID":"8ff3b884-f7ea-4f3a-b70d-7173c3493782","Type":"ContainerDied","Data":"f442755de1def9f46ac76dfc3edfb7870c2ae66f0afbee0dcfe4d34b4f74dbd1"} Sep 11 14:33:47.948485 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.948449 2571 scope.go:117] "RemoveContainer" containerID="12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d" Sep 11 14:33:47.948810 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:33:47.948785 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d\": container with ID starting with 12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d not found: ID does not exist" containerID="12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d" Sep 11 14:33:47.948903 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.948823 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d"} err="failed to get container status \"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d\": rpc error: code = NotFound desc = could not find container \"12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d\": container with ID starting with 12b5ed387314a16e0cde4f8e15bd9352c6f73e152f0a9069f0b594c4b1535a2d not found: ID does not exist" Sep 11 14:33:47.948903 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.948846 2571 scope.go:117] "RemoveContainer" containerID="72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125" Sep 11 14:33:47.980379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.974338 2571 scope.go:117] "RemoveContainer" containerID="72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125" Sep 11 14:33:47.980379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.974864 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:47.983977 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:33:47.983235 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125\": container with ID starting with 72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125 not found: ID does not exist" containerID="72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125" Sep 11 14:33:47.983977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.983278 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125"} err="failed to get container status \"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125\": rpc error: code = NotFound desc = could not find container \"72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125\": container with ID starting with 72a491765608f92548184efae6e06a4359adeed45908f0ad13ee371633e30125 not found: ID does not exist" Sep 11 14:33:47.989584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.989550 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:47.989584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:47.989590 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7"] Sep 11 14:33:48.014559 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.014529 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4jdb\" (UniqueName: \"kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb\") pod \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " Sep 11 14:33:48.014743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.014639 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume\") pod \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\" (UID: \"8ff3b884-f7ea-4f3a-b70d-7173c3493782\") " Sep 11 14:33:48.018112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.015475 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:48.018112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.015509 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lkp\" (UniqueName: \"kubernetes.io/projected/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f-kube-api-access-r6lkp\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:48.018112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.015567 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ff3b884-f7ea-4f3a-b70d-7173c3493782" (UID: "8ff3b884-f7ea-4f3a-b70d-7173c3493782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:48.024511 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.024471 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb" (OuterVolumeSpecName: "kube-api-access-t4jdb") pod "8ff3b884-f7ea-4f3a-b70d-7173c3493782" (UID: "8ff3b884-f7ea-4f3a-b70d-7173c3493782"). InnerVolumeSpecName "kube-api-access-t4jdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:48.116546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.116449 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3b884-f7ea-4f3a-b70d-7173c3493782-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:48.116546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.116483 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jdb\" (UniqueName: \"kubernetes.io/projected/8ff3b884-f7ea-4f3a-b70d-7173c3493782-kube-api-access-t4jdb\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:33:48.122747 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.122711 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:48.267532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.267495 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:48.276066 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.276036 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-93c38c4c-7455779566-vzr8v"] Sep 11 14:33:48.719517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.719422 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-93c38c4c-7cb64bc65c-nbwt7" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.46:9004: i/o timeout" Sep 11 14:33:48.974760 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:48.974667 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.078235 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:49.078198 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.182255 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:49.182214 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.668698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:49.668665 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.959037 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:49.958955 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" path="/var/lib/kubelet/pods/8ff3b884-f7ea-4f3a-b70d-7173c3493782/volumes" Sep 11 14:33:49.959398 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:49.959348 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" path="/var/lib/kubelet/pods/e9882bc3-1bf9-4b0a-b972-f6490ca5c02f/volumes" Sep 11 14:33:50.902010 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:50.901963 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:51.022092 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:51.022061 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:51.270813 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:51.270720 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:52.605712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:52.605677 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:52.708309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:52.708272 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:52.914030 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:52.913925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:53.414425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:53.414383 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:53.858680 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:53.858648 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:54.607530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:54.607486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:56.345874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:56.345835 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:56.913224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:56.913179 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:57.032044 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:57.032005 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:57.775426 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:57.775388 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:58.897254 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:33:58.897211 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.295661 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:00.295619 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.411671 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:00.411628 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.538299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:00.538263 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.132994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.132923 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.260450 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.260401 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.533656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.533624 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:34:01.534075 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534061 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerName="payload-pre-processing" Sep 11 14:34:01.534125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534077 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerName="payload-pre-processing" Sep 11 14:34:01.534125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534088 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" containerName="maas-api" Sep 11 14:34:01.534125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534093 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" containerName="maas-api" Sep 11 14:34:01.534125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534106 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerName="payload-processing" Sep 11 14:34:01.534125 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534112 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerName="payload-processing" Sep 11 14:34:01.534276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534175 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd57bae1-5447-4d8c-b4e8-7cfdeea74e17" containerName="maas-api" Sep 11 14:34:01.534276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534185 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff3b884-f7ea-4f3a-b70d-7173c3493782" containerName="payload-pre-processing" Sep 11 14:34:01.534276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.534192 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9882bc3-1bf9-4b0a-b972-f6490ca5c02f" containerName="payload-processing" Sep 11 14:34:01.538656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.538638 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.540732 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.540701 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-259c79-dockercfg-xzftc\"" Sep 11 14:34:01.540905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.540884 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-259c79\"" Sep 11 14:34:01.553667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.553639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.553788 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.553671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zdz\" (UniqueName: \"kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.556915 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.556891 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:34:01.654446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.654402 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.654446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.654444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zdz\" (UniqueName: \"kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.655176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.655155 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.670306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.670267 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zdz\" (UniqueName: \"kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz\") pod \"payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.767635 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.767590 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.849277 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.849181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:01.975488 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:01.975451 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:02.129486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:02.129382 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:02.161798 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:02.161763 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:34:02.165194 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:34:02.165166 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62b701b_1cf2_4639_83b6_dacf06c077ed.slice/crio-75783cf750353c5b6e022c10444ffa60edde7450f5c1720ebc11677dfc38183c WatchSource:0}: Error finding container 75783cf750353c5b6e022c10444ffa60edde7450f5c1720ebc11677dfc38183c: Status 404 returned error can't find the container with id 75783cf750353c5b6e022c10444ffa60edde7450f5c1720ebc11677dfc38183c Sep 11 14:34:02.718189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:02.718146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:03.016460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:03.016368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" event={"ID":"f62b701b-1cf2-4639-83b6-dacf06c077ed","Type":"ContainerStarted","Data":"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac"} Sep 11 14:34:03.016460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:03.016426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" event={"ID":"f62b701b-1cf2-4639-83b6-dacf06c077ed","Type":"ContainerStarted","Data":"75783cf750353c5b6e022c10444ffa60edde7450f5c1720ebc11677dfc38183c"} Sep 11 14:34:03.016668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:03.016556 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:03.075095 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:03.075043 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" podStartSLOduration=2.07502792 podStartE2EDuration="2.07502792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:03.067233936 +0000 UTC m=+1145.638891429" watchObservedRunningTime="2026-09-11 14:34:03.07502792 +0000 UTC m=+1145.646685392" Sep 11 14:34:14.023959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:14.023900 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:34:23.590154 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:23.590051 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:24.732713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:24.732674 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:25.973192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:25.973146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:27.211745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:27.211709 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:27.341632 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:27.341601 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:28.780276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:28.780241 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:40.725715 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:40.725676 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.282447 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:41.282411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.398116 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:41.398073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.504874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:41.504831 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:48.586293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:48.586256 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:57.582123 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:57.582091 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:57.964004 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:57.963900 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:34:57.967292 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:57.967266 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:34:59.265124 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:34:59.265089 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:00.516008 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:00.515976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:01.543464 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:01.543428 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:02.747475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:02.747436 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:03.731896 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:03.731857 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:03.867625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:03.867567 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:05.863850 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:05.863806 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.055040 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:06.054986 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.212819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:06.212722 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:07.022701 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:07.022666 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:07.996304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:07.996268 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:08.254716 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:08.254617 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:09.318929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:09.318895 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:10.288645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:10.288602 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:11.617174 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:11.617099 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.622624 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:12.622584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.478912 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.478861 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.526727 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.526691 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:35:13.526927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.526901 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" podUID="f62b701b-1cf2-4639-83b6-dacf06c077ed" containerName="payload-pre-processing" containerID="cri-o://4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac" gracePeriod=30 Sep 11 14:35:13.579331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.579294 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.683630 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.683595 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.785288 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.785263 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:35:13.790712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.790678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.884320 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.884281 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.957982 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.957920 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume\") pod \"f62b701b-1cf2-4639-83b6-dacf06c077ed\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " Sep 11 14:35:13.957982 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.957986 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2zdz\" (UniqueName: \"kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz\") pod \"f62b701b-1cf2-4639-83b6-dacf06c077ed\" (UID: \"f62b701b-1cf2-4639-83b6-dacf06c077ed\") " Sep 11 14:35:13.958438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.958412 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f62b701b-1cf2-4639-83b6-dacf06c077ed" (UID: "f62b701b-1cf2-4639-83b6-dacf06c077ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:13.960487 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:13.960459 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz" (OuterVolumeSpecName: "kube-api-access-x2zdz") pod "f62b701b-1cf2-4639-83b6-dacf06c077ed" (UID: "f62b701b-1cf2-4639-83b6-dacf06c077ed"). InnerVolumeSpecName "kube-api-access-x2zdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:14.059192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.059152 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62b701b-1cf2-4639-83b6-dacf06c077ed-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.059192 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.059184 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zdz\" (UniqueName: \"kubernetes.io/projected/f62b701b-1cf2-4639-83b6-dacf06c077ed-kube-api-access-x2zdz\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.244876 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.244844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:14.370550 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.370458 2571 generic.go:356] "Generic (PLEG): container finished" podID="f62b701b-1cf2-4639-83b6-dacf06c077ed" containerID="4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac" exitCode=0 Sep 11 14:35:14.370550 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.370508 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" Sep 11 14:35:14.370550 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.370521 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" event={"ID":"f62b701b-1cf2-4639-83b6-dacf06c077ed","Type":"ContainerDied","Data":"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac"} Sep 11 14:35:14.370796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.370592 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv" event={"ID":"f62b701b-1cf2-4639-83b6-dacf06c077ed","Type":"ContainerDied","Data":"75783cf750353c5b6e022c10444ffa60edde7450f5c1720ebc11677dfc38183c"} Sep 11 14:35:14.370796 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.370616 2571 scope.go:117] "RemoveContainer" containerID="4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac" Sep 11 14:35:14.377067 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.377039 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:14.381603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.381582 2571 scope.go:117] "RemoveContainer" containerID="4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac" Sep 11 14:35:14.382087 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:14.382060 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac\": container with ID starting with 4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac not found: ID does not exist" containerID="4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac" Sep 11 14:35:14.382201 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.382093 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac"} err="failed to get container status \"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac\": rpc error: code = NotFound desc = could not find container \"4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac\": container with ID starting with 4f33a96b034a154544c2d5bbbad7aa8cc357b855b787cff4543b709b496f9bac not found: ID does not exist" Sep 11 14:35:14.397628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.397588 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:35:14.403470 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:14.403442 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-259c79-d7cbfcf9d-5mpjv"] Sep 11 14:35:15.704540 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:15.704501 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:15.959423 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:15.959336 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62b701b-1cf2-4639-83b6-dacf06c077ed" path="/var/lib/kubelet/pods/f62b701b-1cf2-4639-83b6-dacf06c077ed/volumes" Sep 11 14:35:16.680076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:16.680030 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:17.052178 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:17.052141 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:17.597999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:17.597918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.218222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.218188 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:35:18.218696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.218392 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" podUID="febf6857-219e-4e5b-853d-1a193b72a182" containerName="maas-api" containerID="cri-o://80004216d0b1012c756baeb9fd5daa08dd2d2b3e8f610d90ce79e029db2f55c7" gracePeriod=30 Sep 11 14:35:18.248685 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.248647 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.331082 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.330886 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:35:18.331171 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.331146 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" podUID="4b63506a-fb40-4dae-9e71-957c68045261" containerName="payload-processing" containerID="cri-o://2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351" gracePeriod=30 Sep 11 14:35:18.338761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.338708 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:35:18.339048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.339003 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" podUID="4c66155d-ce03-4769-9727-ed8d1c91b30d" containerName="payload-pre-processing" containerID="cri-o://203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003" gracePeriod=30 Sep 11 14:35:18.384738 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.384700 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.397167 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.397141 2571 generic.go:356] "Generic (PLEG): container finished" podID="febf6857-219e-4e5b-853d-1a193b72a182" containerID="80004216d0b1012c756baeb9fd5daa08dd2d2b3e8f610d90ce79e029db2f55c7" exitCode=0 Sep 11 14:35:18.397303 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.397189 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" event={"ID":"febf6857-219e-4e5b-853d-1a193b72a182","Type":"ContainerDied","Data":"80004216d0b1012c756baeb9fd5daa08dd2d2b3e8f610d90ce79e029db2f55c7"} Sep 11 14:35:18.485711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.485633 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.497626 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.497596 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:35:18.592856 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.592824 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.598702 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.598677 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") pod \"febf6857-219e-4e5b-853d-1a193b72a182\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " Sep 11 14:35:18.598848 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.598716 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca\") pod \"febf6857-219e-4e5b-853d-1a193b72a182\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " Sep 11 14:35:18.598848 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.598744 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mc74\" (UniqueName: \"kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74\") pod \"febf6857-219e-4e5b-853d-1a193b72a182\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " Sep 11 14:35:18.598965 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.598856 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs\") pod \"febf6857-219e-4e5b-853d-1a193b72a182\" (UID: \"febf6857-219e-4e5b-853d-1a193b72a182\") " Sep 11 14:35:18.599238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.599189 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "febf6857-219e-4e5b-853d-1a193b72a182" (UID: "febf6857-219e-4e5b-853d-1a193b72a182"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:18.601489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.601454 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "febf6857-219e-4e5b-853d-1a193b72a182" (UID: "febf6857-219e-4e5b-853d-1a193b72a182"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:18.601673 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.601644 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74" (OuterVolumeSpecName: "kube-api-access-2mc74") pod "febf6857-219e-4e5b-853d-1a193b72a182" (UID: "febf6857-219e-4e5b-853d-1a193b72a182"). InnerVolumeSpecName "kube-api-access-2mc74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:18.601793 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.601693 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "febf6857-219e-4e5b-853d-1a193b72a182" (UID: "febf6857-219e-4e5b-853d-1a193b72a182"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:18.610853 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.610737 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:35:18.618592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.618564 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:35:18.700439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700403 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r97t\" (UniqueName: \"kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t\") pod \"4c66155d-ce03-4769-9727-ed8d1c91b30d\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " Sep 11 14:35:18.700614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700485 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume\") pod \"4b63506a-fb40-4dae-9e71-957c68045261\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " Sep 11 14:35:18.700614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700507 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzgjr\" (UniqueName: \"kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr\") pod \"4b63506a-fb40-4dae-9e71-957c68045261\" (UID: \"4b63506a-fb40-4dae-9e71-957c68045261\") " Sep 11 14:35:18.700614 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700534 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume\") pod \"4c66155d-ce03-4769-9727-ed8d1c91b30d\" (UID: \"4c66155d-ce03-4769-9727-ed8d1c91b30d\") " Sep 11 14:35:18.700777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700699 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.700777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700710 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/febf6857-219e-4e5b-853d-1a193b72a182-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.700777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700719 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/febf6857-219e-4e5b-853d-1a193b72a182-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.700777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700730 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc74\" (UniqueName: \"kubernetes.io/projected/febf6857-219e-4e5b-853d-1a193b72a182-kube-api-access-2mc74\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.701025 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.700988 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b63506a-fb40-4dae-9e71-957c68045261" (UID: "4b63506a-fb40-4dae-9e71-957c68045261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:18.701081 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.701019 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c66155d-ce03-4769-9727-ed8d1c91b30d" (UID: "4c66155d-ce03-4769-9727-ed8d1c91b30d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:18.702549 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.702526 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t" (OuterVolumeSpecName: "kube-api-access-5r97t") pod "4c66155d-ce03-4769-9727-ed8d1c91b30d" (UID: "4c66155d-ce03-4769-9727-ed8d1c91b30d"). InnerVolumeSpecName "kube-api-access-5r97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:18.702602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.702569 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr" (OuterVolumeSpecName: "kube-api-access-wzgjr") pod "4b63506a-fb40-4dae-9e71-957c68045261" (UID: "4b63506a-fb40-4dae-9e71-957c68045261"). InnerVolumeSpecName "kube-api-access-wzgjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:18.713345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.713305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.801886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.801847 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r97t\" (UniqueName: \"kubernetes.io/projected/4c66155d-ce03-4769-9727-ed8d1c91b30d-kube-api-access-5r97t\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.801886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.801878 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b63506a-fb40-4dae-9e71-957c68045261-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.801886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.801891 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgjr\" (UniqueName: \"kubernetes.io/projected/4b63506a-fb40-4dae-9e71-957c68045261-kube-api-access-wzgjr\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.802169 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:18.801901 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c66155d-ce03-4769-9727-ed8d1c91b30d-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:19.031287 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.031252 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:19.302626 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.302584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:19.403317 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.403285 2571 generic.go:356] "Generic (PLEG): container finished" podID="4b63506a-fb40-4dae-9e71-957c68045261" containerID="2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351" exitCode=0 Sep 11 14:35:19.403525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.403356 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" Sep 11 14:35:19.403525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.403371 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" event={"ID":"4b63506a-fb40-4dae-9e71-957c68045261","Type":"ContainerDied","Data":"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351"} Sep 11 14:35:19.403525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.403442 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf" event={"ID":"4b63506a-fb40-4dae-9e71-957c68045261","Type":"ContainerDied","Data":"be83551e2d6efd861b97b102585ec35d41f9f2452fdea60b9304c0c71e43704b"} Sep 11 14:35:19.403525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.403468 2571 scope.go:117] "RemoveContainer" containerID="2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351" Sep 11 14:35:19.404927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.404747 2571 generic.go:356] "Generic (PLEG): container finished" podID="4c66155d-ce03-4769-9727-ed8d1c91b30d" containerID="203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003" exitCode=0 Sep 11 14:35:19.404927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.404778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" event={"ID":"4c66155d-ce03-4769-9727-ed8d1c91b30d","Type":"ContainerDied","Data":"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003"} Sep 11 14:35:19.404927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.404814 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" Sep 11 14:35:19.404927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.404826 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x" event={"ID":"4c66155d-ce03-4769-9727-ed8d1c91b30d","Type":"ContainerDied","Data":"ec7e0aaafd4aa14896a09d6fa931d40fe9319b47545d32989d54fd690374c3fd"} Sep 11 14:35:19.406489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.406468 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" Sep 11 14:35:19.406612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.406489 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q" event={"ID":"febf6857-219e-4e5b-853d-1a193b72a182","Type":"ContainerDied","Data":"8591fb83c0b37c61a29fd7aa65798d06717a8b8f1fcbd88fa5535d09f084e68b"} Sep 11 14:35:19.414188 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.414168 2571 scope.go:117] "RemoveContainer" containerID="2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351" Sep 11 14:35:19.414694 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:19.414663 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351\": container with ID starting with 2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351 not found: ID does not exist" containerID="2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351" Sep 11 14:35:19.414787 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.414705 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351"} err="failed to get container status \"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351\": rpc error: code = NotFound desc = could not find container \"2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351\": container with ID starting with 2499348badbdd13ad96024e553613af58285f344dbb2809ad80c61558bd77351 not found: ID does not exist" Sep 11 14:35:19.414787 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.414730 2571 scope.go:117] "RemoveContainer" containerID="203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003" Sep 11 14:35:19.424153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.424132 2571 scope.go:117] "RemoveContainer" containerID="203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003" Sep 11 14:35:19.424475 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:19.424451 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003\": container with ID starting with 203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003 not found: ID does not exist" containerID="203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003" Sep 11 14:35:19.424532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.424485 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003"} err="failed to get container status \"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003\": rpc error: code = NotFound desc = could not find container \"203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003\": container with ID starting with 203e4fe9adf23cbe1a73f63cdbb5c30f5a12022a92c20152709089412aa8d003 not found: ID does not exist" Sep 11 14:35:19.424532 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.424505 2571 scope.go:117] "RemoveContainer" containerID="80004216d0b1012c756baeb9fd5daa08dd2d2b3e8f610d90ce79e029db2f55c7" Sep 11 14:35:19.450297 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.450266 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:35:19.463835 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.463802 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e85961-77d8846ddf-k2v6q"] Sep 11 14:35:19.487294 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.487205 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:35:19.492778 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.492749 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e85961-5c5f6979d9-jpcrf"] Sep 11 14:35:19.516611 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.516578 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:35:19.530119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.530088 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e85961-755d745f8-5s22x"] Sep 11 14:35:19.958586 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.958555 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b63506a-fb40-4dae-9e71-957c68045261" path="/var/lib/kubelet/pods/4b63506a-fb40-4dae-9e71-957c68045261/volumes" Sep 11 14:35:19.958969 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.958956 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c66155d-ce03-4769-9727-ed8d1c91b30d" path="/var/lib/kubelet/pods/4c66155d-ce03-4769-9727-ed8d1c91b30d/volumes" Sep 11 14:35:19.959329 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:19.959317 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="febf6857-219e-4e5b-853d-1a193b72a182" path="/var/lib/kubelet/pods/febf6857-219e-4e5b-853d-1a193b72a182/volumes" Sep 11 14:35:21.094076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:21.094039 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.015157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:24.015123 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.137062 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:24.137028 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.360196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:24.360156 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.798600 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:24.798565 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.915982 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:24.915929 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:25.260689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:25.260605 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:25.731461 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:25.731420 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:26.416667 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:26.416621 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:26.552677 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:26.552639 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:26.675368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:26.675278 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:26.968472 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:26.968367 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:27.095345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:27.095305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:27.289826 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:27.289792 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:27.447645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:27.447611 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:28.226196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:28.226158 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:28.381099 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:28.381066 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:30.226138 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:30.226104 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:30.593625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:30.593588 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:30.696345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:30.696276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:30.924055 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:30.923961 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:31.814563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:31.814509 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:32.655483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:32.655446 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:33.666153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:33.666112 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:33.826080 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:33.826048 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:34.048623 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:34.048589 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:34.220340 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:34.220304 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:34.370378 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:34.370285 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:37.927130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:37.927089 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:38.079896 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:38.079858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:38.667598 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:38.667564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:39.224676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:39.224635 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:42.153577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.153542 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:35:42.154098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.153758 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" podUID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" containerName="maas-api" containerID="cri-o://05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c" gracePeriod=30 Sep 11 14:35:42.214493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.214452 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:42.285343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.285308 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:35:42.285613 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.285580 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" podUID="1cd50313-a735-490f-8d50-35764b4799cf" containerName="payload-pre-processing" containerID="cri-o://a1655b5ed84e122b0eaf16048ae09b9c2a483df6382ab0aff091b7e5c2d80106" gracePeriod=30 Sep 11 14:35:42.421044 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.421019 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:35:42.437128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.437084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:42.516594 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.516555 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4swp4\" (UniqueName: \"kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4\") pod \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " Sep 11 14:35:42.516768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.516624 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls\") pod \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " Sep 11 14:35:42.516768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.516668 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca\") pod \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " Sep 11 14:35:42.516768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.516703 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs\") pod \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\" (UID: \"ca2bc1aa-44ee-4694-9d43-535f21c754f6\") " Sep 11 14:35:42.517114 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.517079 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca2bc1aa-44ee-4694-9d43-535f21c754f6" (UID: "ca2bc1aa-44ee-4694-9d43-535f21c754f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:42.519185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.519157 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca2bc1aa-44ee-4694-9d43-535f21c754f6" (UID: "ca2bc1aa-44ee-4694-9d43-535f21c754f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:42.519373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.519325 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4" (OuterVolumeSpecName: "kube-api-access-4swp4") pod "ca2bc1aa-44ee-4694-9d43-535f21c754f6" (UID: "ca2bc1aa-44ee-4694-9d43-535f21c754f6"). InnerVolumeSpecName "kube-api-access-4swp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:42.519373 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.519327 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca2bc1aa-44ee-4694-9d43-535f21c754f6" (UID: "ca2bc1aa-44ee-4694-9d43-535f21c754f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:42.523610 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.523582 2571 generic.go:356] "Generic (PLEG): container finished" podID="1cd50313-a735-490f-8d50-35764b4799cf" containerID="a1655b5ed84e122b0eaf16048ae09b9c2a483df6382ab0aff091b7e5c2d80106" exitCode=0 Sep 11 14:35:42.523723 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.523606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" event={"ID":"1cd50313-a735-490f-8d50-35764b4799cf","Type":"ContainerDied","Data":"a1655b5ed84e122b0eaf16048ae09b9c2a483df6382ab0aff091b7e5c2d80106"} Sep 11 14:35:42.525058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.525039 2571 generic.go:356] "Generic (PLEG): container finished" podID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" containerID="05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c" exitCode=0 Sep 11 14:35:42.525168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.525066 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" event={"ID":"ca2bc1aa-44ee-4694-9d43-535f21c754f6","Type":"ContainerDied","Data":"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c"} Sep 11 14:35:42.525168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.525084 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" Sep 11 14:35:42.525168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.525093 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf" event={"ID":"ca2bc1aa-44ee-4694-9d43-535f21c754f6","Type":"ContainerDied","Data":"5e59c5f7d7ccae8cba38702e4268b7e235c72346f5e533040871fa28dcf51cba"} Sep 11 14:35:42.525168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.525114 2571 scope.go:117] "RemoveContainer" containerID="05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c" Sep 11 14:35:42.537881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.537859 2571 scope.go:117] "RemoveContainer" containerID="05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c" Sep 11 14:35:42.538302 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:42.538282 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c\": container with ID starting with 05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c not found: ID does not exist" containerID="05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c" Sep 11 14:35:42.538403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.538312 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c"} err="failed to get container status \"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c\": rpc error: code = NotFound desc = could not find container \"05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c\": container with ID starting with 05bfe058928c9f5f7d24883fecb08c5ef033ad9e6c09942b93c76813d3215d5c not found: ID does not exist" Sep 11 14:35:42.541291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.541263 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:42.541656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.541640 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:35:42.563908 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.563871 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:35:42.573770 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.573744 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5a7035-7fbc58c889-ttwgf"] Sep 11 14:35:42.618044 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618013 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume\") pod \"1cd50313-a735-490f-8d50-35764b4799cf\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " Sep 11 14:35:42.618044 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618056 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p2rl\" (UniqueName: \"kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl\") pod \"1cd50313-a735-490f-8d50-35764b4799cf\" (UID: \"1cd50313-a735-490f-8d50-35764b4799cf\") " Sep 11 14:35:42.618361 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618341 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca2bc1aa-44ee-4694-9d43-535f21c754f6-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.618401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618366 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.618401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618382 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swp4\" (UniqueName: \"kubernetes.io/projected/ca2bc1aa-44ee-4694-9d43-535f21c754f6-kube-api-access-4swp4\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.618401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618394 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca2bc1aa-44ee-4694-9d43-535f21c754f6-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.618493 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.618472 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cd50313-a735-490f-8d50-35764b4799cf" (UID: "1cd50313-a735-490f-8d50-35764b4799cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:42.620243 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.620219 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl" (OuterVolumeSpecName: "kube-api-access-5p2rl") pod "1cd50313-a735-490f-8d50-35764b4799cf" (UID: "1cd50313-a735-490f-8d50-35764b4799cf"). InnerVolumeSpecName "kube-api-access-5p2rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:42.627730 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.627692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:42.719910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.719824 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd50313-a735-490f-8d50-35764b4799cf-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.719910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:42.719856 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2rl\" (UniqueName: \"kubernetes.io/projected/1cd50313-a735-490f-8d50-35764b4799cf-kube-api-access-5p2rl\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:35:43.024369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.024276 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:43.249949 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.249901 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:43.530927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.530900 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" Sep 11 14:35:43.531153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.530902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65" event={"ID":"1cd50313-a735-490f-8d50-35764b4799cf","Type":"ContainerDied","Data":"1b5efcd4b0d49117eea34fa461a46cab8c8cc218e13575cae28e34e087203daa"} Sep 11 14:35:43.531153 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.531017 2571 scope.go:117] "RemoveContainer" containerID="a1655b5ed84e122b0eaf16048ae09b9c2a483df6382ab0aff091b7e5c2d80106" Sep 11 14:35:43.567525 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.567495 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:35:43.585203 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.585174 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7035-96ff9c57d-pwt65"] Sep 11 14:35:43.959809 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.959719 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd50313-a735-490f-8d50-35764b4799cf" path="/var/lib/kubelet/pods/1cd50313-a735-490f-8d50-35764b4799cf/volumes" Sep 11 14:35:43.960252 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:43.960230 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" path="/var/lib/kubelet/pods/ca2bc1aa-44ee-4694-9d43-535f21c754f6/volumes" Sep 11 14:35:46.631617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:46.631589 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:46.748403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:46.748367 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:46.920340 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:46.920243 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:47.437292 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:47.437244 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:47.835786 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:47.835746 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:50.472767 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:50.472681 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:50.615486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:50.615451 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:50.784468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:50.784435 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:50.879556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:50.879522 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:51.150569 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:51.150485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:51.716885 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:51.716850 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:51.849770 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:51.849739 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:52.038098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.038066 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:52.226366 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.226327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:52.648774 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.648735 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:52.784916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.784836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:52.889169 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889129 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:35:52.889508 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889496 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c66155d-ce03-4769-9727-ed8d1c91b30d" containerName="payload-pre-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889510 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c66155d-ce03-4769-9727-ed8d1c91b30d" containerName="payload-pre-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889519 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd50313-a735-490f-8d50-35764b4799cf" containerName="payload-pre-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889525 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd50313-a735-490f-8d50-35764b4799cf" containerName="payload-pre-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889538 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b63506a-fb40-4dae-9e71-957c68045261" containerName="payload-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889544 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b63506a-fb40-4dae-9e71-957c68045261" containerName="payload-processing" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889553 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febf6857-219e-4e5b-853d-1a193b72a182" containerName="maas-api" Sep 11 14:35:52.889556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889558 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="febf6857-219e-4e5b-853d-1a193b72a182" containerName="maas-api" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889573 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" containerName="maas-api" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889579 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" containerName="maas-api" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889586 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62b701b-1cf2-4639-83b6-dacf06c077ed" containerName="payload-pre-processing" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889591 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62b701b-1cf2-4639-83b6-dacf06c077ed" containerName="payload-pre-processing" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889639 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b63506a-fb40-4dae-9e71-957c68045261" containerName="payload-processing" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889648 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c66155d-ce03-4769-9727-ed8d1c91b30d" containerName="payload-pre-processing" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889657 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd50313-a735-490f-8d50-35764b4799cf" containerName="payload-pre-processing" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889664 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="febf6857-219e-4e5b-853d-1a193b72a182" containerName="maas-api" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889669 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2bc1aa-44ee-4694-9d43-535f21c754f6" containerName="maas-api" Sep 11 14:35:52.889812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.889674 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62b701b-1cf2-4639-83b6-dacf06c077ed" containerName="payload-pre-processing" Sep 11 14:35:52.892728 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.892710 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:52.897338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.897310 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c3340176\"" Sep 11 14:35:52.897338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.897313 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c3340176-dockercfg-dxrmv\"" Sep 11 14:35:52.910170 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.910141 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:35:52.953489 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.953454 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:35:52.956916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.956894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:52.969529 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:52.969498 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:35:53.019927 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.019889 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s7z\" (UniqueName: \"kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.020126 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.020059 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.121621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.121586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.122035 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.121677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.122131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.122092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xq5k\" (UniqueName: \"kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.122189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.122128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s7z\" (UniqueName: \"kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.122578 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.122545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.132806 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.132777 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s7z\" (UniqueName: \"kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z\") pod \"payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.203059 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.202946 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:53.223255 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.223215 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.223456 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.223299 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xq5k\" (UniqueName: \"kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.223929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.223906 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.236929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.236892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xq5k\" (UniqueName: \"kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k\") pod \"payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.268666 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.268253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:53.514881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.514853 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:35:53.517494 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:35:53.517467 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678742c6_a2c8_485c_814e_b903d485e1b3.slice/crio-72dc950dca09237c2252a51689104c0b0e20bc867ef91fbd1a12a72d5bec98d5 WatchSource:0}: Error finding container 72dc950dca09237c2252a51689104c0b0e20bc867ef91fbd1a12a72d5bec98d5: Status 404 returned error can't find the container with id 72dc950dca09237c2252a51689104c0b0e20bc867ef91fbd1a12a72d5bec98d5 Sep 11 14:35:53.573095 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.573066 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:35:53.575593 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:35:53.575564 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363a3628_9804_4e76_b7c4_f633b9afd285.slice/crio-96737d95c49f4464a80d2cd7b43e9753ef3a3d11577c71d4fdff5b89dc20fbfe WatchSource:0}: Error finding container 96737d95c49f4464a80d2cd7b43e9753ef3a3d11577c71d4fdff5b89dc20fbfe: Status 404 returned error can't find the container with id 96737d95c49f4464a80d2cd7b43e9753ef3a3d11577c71d4fdff5b89dc20fbfe Sep 11 14:35:53.584531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.584400 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" event={"ID":"363a3628-9804-4e76-b7c4-f633b9afd285","Type":"ContainerStarted","Data":"96737d95c49f4464a80d2cd7b43e9753ef3a3d11577c71d4fdff5b89dc20fbfe"} Sep 11 14:35:53.586089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:53.586062 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" event={"ID":"678742c6-a2c8-485c-814e-b903d485e1b3","Type":"ContainerStarted","Data":"72dc950dca09237c2252a51689104c0b0e20bc867ef91fbd1a12a72d5bec98d5"} Sep 11 14:35:54.592357 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.592323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" event={"ID":"678742c6-a2c8-485c-814e-b903d485e1b3","Type":"ContainerStarted","Data":"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48"} Sep 11 14:35:54.592843 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.592392 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:35:54.594010 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.593981 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" event={"ID":"363a3628-9804-4e76-b7c4-f633b9afd285","Type":"ContainerStarted","Data":"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa"} Sep 11 14:35:54.594133 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.594028 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:35:54.630603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.630557 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" podStartSLOduration=2.630542262 podStartE2EDuration="2.630542262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:54.628107983 +0000 UTC m=+1257.199765509" watchObservedRunningTime="2026-09-11 14:35:54.630542262 +0000 UTC m=+1257.202199735" Sep 11 14:35:54.660089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:54.660038 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" podStartSLOduration=2.660021661 podStartE2EDuration="2.660021661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:54.658693252 +0000 UTC m=+1257.230350745" watchObservedRunningTime="2026-09-11 14:35:54.660021661 +0000 UTC m=+1257.231679134" Sep 11 14:35:55.114210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:55.114178 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:55.357866 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:55.357828 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:55.530952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:55.530905 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:56.892360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:56.892308 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:57.465781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:57.465744 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:58.846156 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.846123 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:35:58.849592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.849569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.853016 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.852991 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-09b16c-serving-cert\"" Sep 11 14:35:58.863689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.863656 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:35:58.874903 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.874871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.875079 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.874979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.875079 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.875037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9s4\" (UniqueName: \"kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.875180 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.875128 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.916610 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.916570 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:58.976318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.976277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.976499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.976337 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.976499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.976388 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9s4\" (UniqueName: \"kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.976656 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:58.976495 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-serving-cert: secret "maas-api-e2e-shared-a-w3-09b16c-serving-cert" not found Sep 11 14:35:58.976656 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:35:58.976565 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls podName:36764d3d-19d0-470e-8eb9-640832ffd114 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:59.476545618 +0000 UTC m=+1262.048203070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls") pod "maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" (UID: "36764d3d-19d0-470e-8eb9-640832ffd114") : secret "maas-api-e2e-shared-a-w3-09b16c-serving-cert" not found Sep 11 14:35:58.976656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.976644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.977351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.977326 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.979305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.979281 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:58.990328 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:58.990302 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9s4\" (UniqueName: \"kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:59.019606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.019571 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:59.341384 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.341346 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:59.480197 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.480159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:59.482530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.482499 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:59.761754 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.761651 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:35:59.786056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.786024 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:59.947391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:35:59.947358 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:00.056260 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.056227 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:36:00.058117 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:36:00.058085 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36764d3d_19d0_470e_8eb9_640832ffd114.slice/crio-aa60ee6812a3da7211a9236045c7d3250e46e89afb9a3ff8628b5dc5bebdc62d WatchSource:0}: Error finding container aa60ee6812a3da7211a9236045c7d3250e46e89afb9a3ff8628b5dc5bebdc62d: Status 404 returned error can't find the container with id aa60ee6812a3da7211a9236045c7d3250e46e89afb9a3ff8628b5dc5bebdc62d Sep 11 14:36:00.059911 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.059894 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:36:00.120295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.120258 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:36:00.125226 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.125201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.127164 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:00.127135 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.127363 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:00.127343 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-shared-a-w3-09b16c\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-09b16c\"" type="*v1.ConfigMap" Sep 11 14:36:00.127414 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:00.127363 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-shared-a-w3-09b16c-dockercfg-nswlm\" is forbidden: User \"system:node:ip-10-0-141-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-09b16c-dockercfg-nswlm\"" type="*v1.Secret" Sep 11 14:36:00.148628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.148600 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:36:00.186152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.186119 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xqz\" (UniqueName: \"kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.186152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.186152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.287574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.287532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xqz\" (UniqueName: \"kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.287574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.287573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.310326 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.310249 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xqz\" (UniqueName: \"kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:00.624113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.624030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" event={"ID":"36764d3d-19d0-470e-8eb9-640832ffd114","Type":"ContainerStarted","Data":"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc"} Sep 11 14:36:00.624113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.624084 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" event={"ID":"36764d3d-19d0-470e-8eb9-640832ffd114","Type":"ContainerStarted","Data":"aa60ee6812a3da7211a9236045c7d3250e46e89afb9a3ff8628b5dc5bebdc62d"} Sep 11 14:36:00.624285 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.624151 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:36:00.646812 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.646775 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:00.736014 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:00.735953 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" podStartSLOduration=2.335013615 podStartE2EDuration="2.735918148s" totalImagesPullingTime="400.904533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:00.733340527 +0000 UTC m=+1263.304998027" watchObservedRunningTime="2026-09-11 14:36:00.735918148 +0000 UTC m=+1263.307575620" Sep 11 14:36:01.289515 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:01.289477 2571 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-a-w3-09b16c: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:36:01.289926 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:01.289569 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume podName:c9f3b930-07dd-47a0-bc4c-37896d0c305a nodeName:}" failed. No retries permitted until 2026-09-11 14:36:01.789547085 +0000 UTC m=+1264.361204552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume") pod "payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" (UID: "c9f3b930-07dd-47a0-bc4c-37896d0c305a") : failed to sync configmap cache: timed out waiting for the condition Sep 11 14:36:01.389835 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:01.389803 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-09b16c-dockercfg-nswlm\"" Sep 11 14:36:01.672431 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:01.672350 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-09b16c\"" Sep 11 14:36:01.800828 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:01.800794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:01.801507 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:01.801489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:01.936176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:01.936079 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:02.226460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.226433 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:36:02.229796 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:36:02.229749 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f3b930_07dd_47a0_bc4c_37896d0c305a.slice/crio-e3bc39c7eb6519970f5a613ec357cce216f57bedf77cff9679348d17afe0ea0e WatchSource:0}: Error finding container e3bc39c7eb6519970f5a613ec357cce216f57bedf77cff9679348d17afe0ea0e: Status 404 returned error can't find the container with id e3bc39c7eb6519970f5a613ec357cce216f57bedf77cff9679348d17afe0ea0e Sep 11 14:36:02.335628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.335594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:02.635457 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.635427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" event={"ID":"c9f3b930-07dd-47a0-bc4c-37896d0c305a","Type":"ContainerStarted","Data":"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4"} Sep 11 14:36:02.635649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.635475 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" event={"ID":"c9f3b930-07dd-47a0-bc4c-37896d0c305a","Type":"ContainerStarted","Data":"e3bc39c7eb6519970f5a613ec357cce216f57bedf77cff9679348d17afe0ea0e"} Sep 11 14:36:02.635649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.635577 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:02.679995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:02.679924 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" podStartSLOduration=2.679907454 podStartE2EDuration="2.679907454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:02.678584604 +0000 UTC m=+1265.250242077" watchObservedRunningTime="2026-09-11 14:36:02.679907454 +0000 UTC m=+1265.251564926" Sep 11 14:36:05.601264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:05.601227 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:36:05.601775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:05.601400 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:36:06.634765 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:06.634734 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:36:13.264757 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:13.264711 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:13.642348 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:13.642313 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:36:15.428626 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:15.428586 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:15.563711 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:15.563675 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:16.362113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:16.362074 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:16.459774 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:16.459737 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:16.566618 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:16.566574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:16.836825 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:16.836790 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:17.967981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:17.967930 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:18.684083 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:18.684049 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:19.125280 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:19.125245 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:20.130195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:20.130160 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:20.361416 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:20.361386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:20.755090 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:20.755052 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:21.123952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:21.123902 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:21.238574 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:21.238538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:21.347637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:21.347595 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:22.174106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:22.174071 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:23.882364 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:23.882327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:24.583840 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:24.583788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:24.682713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:24.682678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:25.337776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:25.337738 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:25.680734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:25.680644 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:25.784597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:25.784543 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:26.840185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:26.840116 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.032636 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.032595 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.130994 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.130929 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.230445 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.230411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.694237 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.694204 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.785751 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.785718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:28.898826 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.898787 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:36:28.901809 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.901786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:28.903703 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.903681 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1c284e\"" Sep 11 14:36:28.903835 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.903734 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1c284e-dockercfg-z7wth\"" Sep 11 14:36:28.917733 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.917703 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:36:28.982162 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.982080 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:36:28.985872 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:28.985849 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.000861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.000835 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:36:29.069332 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.069299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.069709 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.069345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9zw\" (UniqueName: \"kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.069709 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.069427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.069709 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.069502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs7f\" (UniqueName: \"kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.170957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.170903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.171152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.171004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9zw\" (UniqueName: \"kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.171152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.171060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.171152 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.171100 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs7f\" (UniqueName: \"kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.171656 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.171630 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.171827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.171807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.188085 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.188057 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs7f\" (UniqueName: \"kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f\") pod \"payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.188248 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.188224 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9zw\" (UniqueName: \"kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw\") pod \"payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.212174 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.212144 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.258798 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.258673 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:29.296106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.296068 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.494970 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.494923 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:36:29.497220 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:36:29.497194 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffd3cb9_bf29_4d80_93d4_e281986451b6.slice/crio-c91f882f4bf9640bb0de35fc8d46655f73127be972eaebdb71279ef573312370 WatchSource:0}: Error finding container c91f882f4bf9640bb0de35fc8d46655f73127be972eaebdb71279ef573312370: Status 404 returned error can't find the container with id c91f882f4bf9640bb0de35fc8d46655f73127be972eaebdb71279ef573312370 Sep 11 14:36:29.599212 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.597362 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:36:29.605099 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:36:29.605068 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f49c792_6069_4ea0_bd8b_e783111ae550.slice/crio-1f03fd96c3dee83a252cbec84e4f972f8bd1d8386443f071b6673fb86b3a3029 WatchSource:0}: Error finding container 1f03fd96c3dee83a252cbec84e4f972f8bd1d8386443f071b6673fb86b3a3029: Status 404 returned error can't find the container with id 1f03fd96c3dee83a252cbec84e4f972f8bd1d8386443f071b6673fb86b3a3029 Sep 11 14:36:29.781285 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.781241 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" event={"ID":"8f49c792-6069-4ea0-bd8b-e783111ae550","Type":"ContainerStarted","Data":"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace"} Sep 11 14:36:29.781522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.781323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" event={"ID":"8f49c792-6069-4ea0-bd8b-e783111ae550","Type":"ContainerStarted","Data":"1f03fd96c3dee83a252cbec84e4f972f8bd1d8386443f071b6673fb86b3a3029"} Sep 11 14:36:29.781522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.781378 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:29.783701 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.783668 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" event={"ID":"3ffd3cb9-bf29-4d80-93d4-e281986451b6","Type":"ContainerStarted","Data":"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c"} Sep 11 14:36:29.783839 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.783722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" event={"ID":"3ffd3cb9-bf29-4d80-93d4-e281986451b6","Type":"ContainerStarted","Data":"c91f882f4bf9640bb0de35fc8d46655f73127be972eaebdb71279ef573312370"} Sep 11 14:36:29.783839 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.783748 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:29.810335 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.810273 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" podStartSLOduration=1.810253689 podStartE2EDuration="1.810253689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:29.807388459 +0000 UTC m=+1292.379045960" watchObservedRunningTime="2026-09-11 14:36:29.810253689 +0000 UTC m=+1292.381911155" Sep 11 14:36:29.829268 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:29.829202 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" podStartSLOduration=1.82917873 podStartE2EDuration="1.82917873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:29.828601124 +0000 UTC m=+1292.400258619" watchObservedRunningTime="2026-09-11 14:36:29.82917873 +0000 UTC m=+1292.400836204" Sep 11 14:36:30.344492 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:30.344455 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:31.411881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:31.411839 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:36.373190 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:36.373151 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.205282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.205204 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.259161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.259126 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:36:37.259412 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.259382 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" podUID="363a3628-9804-4e76-b7c4-f633b9afd285" containerName="payload-processing" containerID="cri-o://b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa" gracePeriod=30 Sep 11 14:36:37.269628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.269601 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:36:37.269864 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.269836 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" podUID="678742c6-a2c8-485c-814e-b903d485e1b3" containerName="payload-pre-processing" containerID="cri-o://80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48" gracePeriod=30 Sep 11 14:36:37.307736 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.307698 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.406983 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.406814 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.507335 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.507294 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.548802 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.548777 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:36:37.551522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.551502 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:36:37.556087 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556061 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume\") pod \"363a3628-9804-4e76-b7c4-f633b9afd285\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " Sep 11 14:36:37.556218 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556089 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume\") pod \"678742c6-a2c8-485c-814e-b903d485e1b3\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " Sep 11 14:36:37.556218 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556151 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xq5k\" (UniqueName: \"kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k\") pod \"363a3628-9804-4e76-b7c4-f633b9afd285\" (UID: \"363a3628-9804-4e76-b7c4-f633b9afd285\") " Sep 11 14:36:37.556218 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556170 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6s7z\" (UniqueName: \"kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z\") pod \"678742c6-a2c8-485c-814e-b903d485e1b3\" (UID: \"678742c6-a2c8-485c-814e-b903d485e1b3\") " Sep 11 14:36:37.556486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556464 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "363a3628-9804-4e76-b7c4-f633b9afd285" (UID: "363a3628-9804-4e76-b7c4-f633b9afd285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:36:37.556556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.556526 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "678742c6-a2c8-485c-814e-b903d485e1b3" (UID: "678742c6-a2c8-485c-814e-b903d485e1b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:36:37.558420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.558397 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z" (OuterVolumeSpecName: "kube-api-access-k6s7z") pod "678742c6-a2c8-485c-814e-b903d485e1b3" (UID: "678742c6-a2c8-485c-814e-b903d485e1b3"). InnerVolumeSpecName "kube-api-access-k6s7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:37.559543 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.559520 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k" (OuterVolumeSpecName: "kube-api-access-2xq5k") pod "363a3628-9804-4e76-b7c4-f633b9afd285" (UID: "363a3628-9804-4e76-b7c4-f633b9afd285"). InnerVolumeSpecName "kube-api-access-2xq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:37.585521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.585485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.657357 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.657324 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363a3628-9804-4e76-b7c4-f633b9afd285-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.657357 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.657351 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678742c6-a2c8-485c-814e-b903d485e1b3-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.657357 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.657362 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xq5k\" (UniqueName: \"kubernetes.io/projected/363a3628-9804-4e76-b7c4-f633b9afd285-kube-api-access-2xq5k\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.657603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.657371 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s7z\" (UniqueName: \"kubernetes.io/projected/678742c6-a2c8-485c-814e-b903d485e1b3-kube-api-access-k6s7z\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.826768 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.826735 2571 generic.go:356] "Generic (PLEG): container finished" podID="678742c6-a2c8-485c-814e-b903d485e1b3" containerID="80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48" exitCode=0 Sep 11 14:36:37.827143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.826774 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" Sep 11 14:36:37.827143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.826822 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" event={"ID":"678742c6-a2c8-485c-814e-b903d485e1b3","Type":"ContainerDied","Data":"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48"} Sep 11 14:36:37.827143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.826875 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q" event={"ID":"678742c6-a2c8-485c-814e-b903d485e1b3","Type":"ContainerDied","Data":"72dc950dca09237c2252a51689104c0b0e20bc867ef91fbd1a12a72d5bec98d5"} Sep 11 14:36:37.827143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.826900 2571 scope.go:117] "RemoveContainer" containerID="80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48" Sep 11 14:36:37.828313 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.828290 2571 generic.go:356] "Generic (PLEG): container finished" podID="363a3628-9804-4e76-b7c4-f633b9afd285" containerID="b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa" exitCode=0 Sep 11 14:36:37.828417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.828333 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" Sep 11 14:36:37.828417 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.828381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" event={"ID":"363a3628-9804-4e76-b7c4-f633b9afd285","Type":"ContainerDied","Data":"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa"} Sep 11 14:36:37.828531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.828436 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx" event={"ID":"363a3628-9804-4e76-b7c4-f633b9afd285","Type":"ContainerDied","Data":"96737d95c49f4464a80d2cd7b43e9753ef3a3d11577c71d4fdff5b89dc20fbfe"} Sep 11 14:36:37.837546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.837526 2571 scope.go:117] "RemoveContainer" containerID="80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48" Sep 11 14:36:37.837881 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:37.837853 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48\": container with ID starting with 80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48 not found: ID does not exist" containerID="80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48" Sep 11 14:36:37.838028 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.837889 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48"} err="failed to get container status \"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48\": rpc error: code = NotFound desc = could not find container \"80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48\": container with ID starting with 80f92be57cd444a905410abf1394c76cf041b627872f567c1f43fbce367b8a48 not found: ID does not exist" Sep 11 14:36:37.838028 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.837910 2571 scope.go:117] "RemoveContainer" containerID="b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa" Sep 11 14:36:37.847586 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.847568 2571 scope.go:117] "RemoveContainer" containerID="b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa" Sep 11 14:36:37.847866 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:36:37.847844 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa\": container with ID starting with b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa not found: ID does not exist" containerID="b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa" Sep 11 14:36:37.848002 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.847870 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa"} err="failed to get container status \"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa\": rpc error: code = NotFound desc = could not find container \"b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa\": container with ID starting with b70a9d786ca6ff0fe99b1bf3a7f222dbcacf2d81b8bad3a62729a8ed740935aa not found: ID does not exist" Sep 11 14:36:37.851776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.851750 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:36:37.860179 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.860155 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c3340176-5ccd8bdb5d-vnq2q"] Sep 11 14:36:37.877202 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.877172 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:36:37.882527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.882503 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c3340176-77d6b4fbf-4r5nx"] Sep 11 14:36:37.958283 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.958246 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.959892 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.959864 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363a3628-9804-4e76-b7c4-f633b9afd285" path="/var/lib/kubelet/pods/363a3628-9804-4e76-b7c4-f633b9afd285/volumes" Sep 11 14:36:37.960299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:37.960282 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="678742c6-a2c8-485c-814e-b903d485e1b3" path="/var/lib/kubelet/pods/678742c6-a2c8-485c-814e-b903d485e1b3/volumes" Sep 11 14:36:38.057394 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:38.057362 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:39.148806 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:39.148765 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:40.306698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:40.306666 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:40.793143 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:40.793106 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:36:40.793324 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:40.793287 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:36:42.511684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:42.511648 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:43.813043 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:43.813010 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:44.489038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:44.489006 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:45.349634 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:45.349597 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:46.681913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:46.681877 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:46.765126 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:46.765095 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:47.051196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:47.051153 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:47.576076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:47.576046 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:47.667918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:47.667886 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:47.818620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:47.818587 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:54.638695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:54.638657 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:56.345241 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:56.345201 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:56.424035 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:56.423996 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:56.529916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:56.529887 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:56.669743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:56.669648 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:56.768279 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:36:56.768236 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:04.555371 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:04.555329 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:04.641036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:04.640994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:05.727078 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:05.727038 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:05.928610 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:05.928563 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:06.125540 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:06.125506 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:06.214874 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:06.214836 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:07.220579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:07.220547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:08.551437 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:08.551402 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:09.129597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:09.129554 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:09.231376 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:09.231343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:09.862714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:09.862683 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:11.364290 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:11.364252 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:12.610700 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:12.610662 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:12.706851 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:12.706809 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:13.115545 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.115505 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:13.316130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.316095 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:13.417829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.417746 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:13.511127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511092 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:37:13.511505 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511492 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363a3628-9804-4e76-b7c4-f633b9afd285" containerName="payload-processing" Sep 11 14:37:13.511548 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511509 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="363a3628-9804-4e76-b7c4-f633b9afd285" containerName="payload-processing" Sep 11 14:37:13.511548 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511537 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678742c6-a2c8-485c-814e-b903d485e1b3" containerName="payload-pre-processing" Sep 11 14:37:13.511548 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511543 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="678742c6-a2c8-485c-814e-b903d485e1b3" containerName="payload-pre-processing" Sep 11 14:37:13.511648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511597 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="678742c6-a2c8-485c-814e-b903d485e1b3" containerName="payload-pre-processing" Sep 11 14:37:13.511648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.511607 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="363a3628-9804-4e76-b7c4-f633b9afd285" containerName="payload-processing" Sep 11 14:37:13.516392 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.516375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.518446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.518424 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-958398-dockercfg-v5km5\"" Sep 11 14:37:13.518446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.518441 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-958398\"" Sep 11 14:37:13.530358 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.530337 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:37:13.585281 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.585247 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:37:13.589006 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.588990 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.605015 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.604988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:37:13.608867 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.608842 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.609037 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.608891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6qz\" (UniqueName: \"kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.710442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.710363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.710442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.710415 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6qz\" (UniqueName: \"kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.710442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.710441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.710862 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.710506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fppg\" (UniqueName: \"kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.711105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.711086 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.721434 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.721406 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6qz\" (UniqueName: \"kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz\") pod \"payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.811207 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.811169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.811375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.811256 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fppg\" (UniqueName: \"kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.811980 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.811961 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.821840 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.821815 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fppg\" (UniqueName: \"kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg\") pod \"payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:13.826786 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.826757 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:13.899319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:13.899287 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:14.110250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:14.110226 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:37:14.112111 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:37:14.112083 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode386cbc9_5494_4464_be94_83b163194f6d.slice/crio-d31e6b90165995f9ec2e3a4a841d2857485c00be67cf807868924454a08eac70 WatchSource:0}: Error finding container d31e6b90165995f9ec2e3a4a841d2857485c00be67cf807868924454a08eac70: Status 404 returned error can't find the container with id d31e6b90165995f9ec2e3a4a841d2857485c00be67cf807868924454a08eac70 Sep 11 14:37:14.199641 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:14.199620 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:37:14.201759 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:37:14.201733 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0bb9bb_90cc_4b78_880f_cf736a65b8c8.slice/crio-393eaf6415406caca0c5574d1c03e24c753acf07c1bc637d0b1ff3e66390d988 WatchSource:0}: Error finding container 393eaf6415406caca0c5574d1c03e24c753acf07c1bc637d0b1ff3e66390d988: Status 404 returned error can't find the container with id 393eaf6415406caca0c5574d1c03e24c753acf07c1bc637d0b1ff3e66390d988 Sep 11 14:37:15.004865 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.004812 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" event={"ID":"e386cbc9-5494-4464-be94-83b163194f6d","Type":"ContainerStarted","Data":"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3"} Sep 11 14:37:15.005394 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.005069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:15.005394 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.005109 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" event={"ID":"e386cbc9-5494-4464-be94-83b163194f6d","Type":"ContainerStarted","Data":"d31e6b90165995f9ec2e3a4a841d2857485c00be67cf807868924454a08eac70"} Sep 11 14:37:15.006680 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.006658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" event={"ID":"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8","Type":"ContainerStarted","Data":"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885"} Sep 11 14:37:15.006807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.006700 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" event={"ID":"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8","Type":"ContainerStarted","Data":"393eaf6415406caca0c5574d1c03e24c753acf07c1bc637d0b1ff3e66390d988"} Sep 11 14:37:15.006807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.006741 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:15.042239 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.042177 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" podStartSLOduration=2.042158598 podStartE2EDuration="2.042158598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:15.040817034 +0000 UTC m=+1337.612474533" watchObservedRunningTime="2026-09-11 14:37:15.042158598 +0000 UTC m=+1337.613816072" Sep 11 14:37:15.070772 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:15.070718 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" podStartSLOduration=2.070702647 podStartE2EDuration="2.070702647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:15.069578601 +0000 UTC m=+1337.641236094" watchObservedRunningTime="2026-09-11 14:37:15.070702647 +0000 UTC m=+1337.642360159" Sep 11 14:37:26.013469 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:26.013433 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:37:26.013964 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:26.013519 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:37:28.605022 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:28.604989 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:35.945243 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:35.945209 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:37.028286 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:37.028250 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:38.315040 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:38.315000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:39.276917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:39.276873 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:39.465379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:39.465340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:39.684051 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:39.683963 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:40.077056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:40.077019 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:40.519383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:40.519294 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:42.585835 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:42.585795 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:43.163165 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:43.163132 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:43.279567 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:43.279532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:43.851032 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:43.850905 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:45.226684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:45.226632 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:46.587291 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:46.587254 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:47.826314 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:47.826269 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:47.928360 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:47.928328 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:48.151544 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.151457 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:48.717543 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.717513 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:48.811183 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.811125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:48.820123 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.820095 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:37:48.823881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.823863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:48.826069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.826048 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-455f2e\"" Sep 11 14:37:48.826193 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.826175 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-455f2e-dockercfg-szvql\"" Sep 11 14:37:48.844305 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.844278 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:37:48.950349 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.950310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fzr\" (UniqueName: \"kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:48.950521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:48.950375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.051849 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.051807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.052072 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.051923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fzr\" (UniqueName: \"kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.052589 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.052570 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.063982 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.063956 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fzr\" (UniqueName: \"kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr\") pod \"payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.135005 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.134967 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:49.421189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:49.421162 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:37:49.424283 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:37:49.424243 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d9c982_9d32_438b_98b6_4227d39d838e.slice/crio-07c8ad184137dabf09226f1f00d286759290e2dd9f7d419da37038a981746a4c WatchSource:0}: Error finding container 07c8ad184137dabf09226f1f00d286759290e2dd9f7d419da37038a981746a4c: Status 404 returned error can't find the container with id 07c8ad184137dabf09226f1f00d286759290e2dd9f7d419da37038a981746a4c Sep 11 14:37:50.176425 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:50.176387 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" event={"ID":"a3d9c982-9d32-438b-98b6-4227d39d838e","Type":"ContainerStarted","Data":"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7"} Sep 11 14:37:50.176823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:50.176443 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" event={"ID":"a3d9c982-9d32-438b-98b6-4227d39d838e","Type":"ContainerStarted","Data":"07c8ad184137dabf09226f1f00d286759290e2dd9f7d419da37038a981746a4c"} Sep 11 14:37:50.176823 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:50.176534 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:37:50.218246 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:37:50.218194 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" podStartSLOduration=2.218178885 podStartE2EDuration="2.218178885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:50.216880524 +0000 UTC m=+1372.788538023" watchObservedRunningTime="2026-09-11 14:37:50.218178885 +0000 UTC m=+1372.789836357" Sep 11 14:38:01.184229 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:01.184193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:38:11.021652 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:11.021617 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:12.303558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:12.303516 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:13.370455 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:13.370416 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:13.465900 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:13.465866 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:14.670713 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:14.670670 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:15.891442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:15.891407 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:17.846442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:17.846406 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:23.055913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:23.055879 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:23.266671 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:23.266637 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:23.605620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:23.605584 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:25.760775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:25.760736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:26.488867 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:26.488833 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:26.575222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:26.575185 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:26.992112 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:26.992079 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:28.308318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:28.308284 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:29.529251 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:29.529219 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:30.808769 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:30.808731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:32.007084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.007043 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:38:32.010743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.010712 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.012706 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.012685 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b947f70-serving-cert\"" Sep 11 14:38:32.037060 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.037030 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:38:32.046917 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.046878 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:32.123696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.123659 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:32.166243 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.166205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22k9j\" (UniqueName: \"kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.166413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.166287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.166413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.166312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.166413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.166355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.267617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.267526 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.267617 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.267593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22k9j\" (UniqueName: \"kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.267870 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.267654 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.267870 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.267680 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.268378 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.268356 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.269992 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.269969 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.270084 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.270050 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.277336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.277306 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22k9j\" (UniqueName: \"kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j\") pod \"maas-api-e2e-mt-2b947f70-99d75b956-6k9hr\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.320906 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.320878 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:32.321606 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.321576 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:32.607628 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.607604 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:38:32.609892 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:38:32.609857 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec07bae6_0b99_44c2_a58e_708366cdf86a.slice/crio-d8af9cfd682ee6ecaf99d47db3660238cde0f7840e3739b9fa07ff3c01a3bd7f WatchSource:0}: Error finding container d8af9cfd682ee6ecaf99d47db3660238cde0f7840e3739b9fa07ff3c01a3bd7f: Status 404 returned error can't find the container with id d8af9cfd682ee6ecaf99d47db3660238cde0f7840e3739b9fa07ff3c01a3bd7f Sep 11 14:38:32.833440 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.833405 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:32.944230 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:32.944135 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:33.103427 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.103389 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:38:33.108136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.108115 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.110266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.110243 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b947f70-dockercfg-2pj44\"" Sep 11 14:38:33.110663 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.110648 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b947f70\"" Sep 11 14:38:33.126160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.126119 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:38:33.278528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.278484 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6w4\" (UniqueName: \"kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.278528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.278524 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.379269 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.379230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6w4\" (UniqueName: \"kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.379269 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.379267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.379949 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.379920 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.391111 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.391080 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6w4\" (UniqueName: \"kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4\") pod \"payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.405424 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.405384 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" event={"ID":"ec07bae6-0b99-44c2-a58e-708366cdf86a","Type":"ContainerStarted","Data":"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f"} Sep 11 14:38:33.405621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.405453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" event={"ID":"ec07bae6-0b99-44c2-a58e-708366cdf86a","Type":"ContainerStarted","Data":"d8af9cfd682ee6ecaf99d47db3660238cde0f7840e3739b9fa07ff3c01a3bd7f"} Sep 11 14:38:33.405621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.405501 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:33.419696 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.419665 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:33.435785 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.435708 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" podStartSLOduration=1.851301522 podStartE2EDuration="2.435686795s" totalImagesPullingTime="584.385273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:33.432573544 +0000 UTC m=+1416.004231037" watchObservedRunningTime="2026-09-11 14:38:33.435686795 +0000 UTC m=+1416.007344269" Sep 11 14:38:33.754430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:33.754398 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:38:33.756906 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:38:33.756880 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0899440f_6b0e_465e_a3db_94518b0e7bb5.slice/crio-34cc0a5e750c0bd38c5c6aa29a9a495aa30cfa06adccb91337c94b34f2625d32 WatchSource:0}: Error finding container 34cc0a5e750c0bd38c5c6aa29a9a495aa30cfa06adccb91337c94b34f2625d32: Status 404 returned error can't find the container with id 34cc0a5e750c0bd38c5c6aa29a9a495aa30cfa06adccb91337c94b34f2625d32 Sep 11 14:38:34.412393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:34.412349 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" event={"ID":"0899440f-6b0e-465e-a3db-94518b0e7bb5","Type":"ContainerStarted","Data":"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979"} Sep 11 14:38:34.412860 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:34.412443 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" event={"ID":"0899440f-6b0e-465e-a3db-94518b0e7bb5","Type":"ContainerStarted","Data":"34cc0a5e750c0bd38c5c6aa29a9a495aa30cfa06adccb91337c94b34f2625d32"} Sep 11 14:38:34.412860 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:34.412525 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:34.449929 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:34.449875 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" podStartSLOduration=1.449859362 podStartE2EDuration="1.449859362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:34.447479755 +0000 UTC m=+1417.019137230" watchObservedRunningTime="2026-09-11 14:38:34.449859362 +0000 UTC m=+1417.021516835" Sep 11 14:38:39.418517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:39.418482 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:38:45.420587 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:45.420555 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:38:55.451482 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:55.451401 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:56.732589 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:56.732554 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:56.822734 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:56.822696 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:58.082573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:58.082537 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:59.192952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:38:59.192899 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:00.378391 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:00.378355 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:01.494053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:01.494020 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:10.068395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:10.068353 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:53.968126 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:53.968092 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:55.110495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:55.110449 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:56.277642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:56.277608 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:57.298876 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:57.298840 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:57.994834 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:57.994807 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:39:58.003884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:58.003859 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:39:58.513583 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:58.513537 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:59.611988 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:39:59.611929 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:03.538947 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:03.538908 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:04.731554 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:04.731518 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.224483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.224449 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:40:09.225040 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.224687 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerName="maas-api" containerID="cri-o://3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f" gracePeriod=30 Sep 11 14:40:09.268863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.268835 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.315315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.315285 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:40:09.315536 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.315509 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" podUID="0899440f-6b0e-465e-a3db-94518b0e7bb5" containerName="payload-processing" containerID="cri-o://22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979" gracePeriod=30 Sep 11 14:40:09.386497 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.386467 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.413668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.413637 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": dial tcp 10.133.0.57:8443: connect: connection refused" Sep 11 14:40:09.480304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.480207 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.598909 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.598655 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:40:09.599437 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.599391 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.623799 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.623778 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:40:09.698207 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv6w4\" (UniqueName: \"kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4\") pod \"0899440f-6b0e-465e-a3db-94518b0e7bb5\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " Sep 11 14:40:09.698401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698217 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume\") pod \"0899440f-6b0e-465e-a3db-94518b0e7bb5\" (UID: \"0899440f-6b0e-465e-a3db-94518b0e7bb5\") " Sep 11 14:40:09.698401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698253 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca\") pod \"ec07bae6-0b99-44c2-a58e-708366cdf86a\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " Sep 11 14:40:09.698401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698271 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22k9j\" (UniqueName: \"kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j\") pod \"ec07bae6-0b99-44c2-a58e-708366cdf86a\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " Sep 11 14:40:09.698401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698298 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls\") pod \"ec07bae6-0b99-44c2-a58e-708366cdf86a\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " Sep 11 14:40:09.698401 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698356 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs\") pod \"ec07bae6-0b99-44c2-a58e-708366cdf86a\" (UID: \"ec07bae6-0b99-44c2-a58e-708366cdf86a\") " Sep 11 14:40:09.698744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698668 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec07bae6-0b99-44c2-a58e-708366cdf86a" (UID: "ec07bae6-0b99-44c2-a58e-708366cdf86a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:40:09.698744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.698718 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0899440f-6b0e-465e-a3db-94518b0e7bb5" (UID: "0899440f-6b0e-465e-a3db-94518b0e7bb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:40:09.700642 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.700610 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4" (OuterVolumeSpecName: "kube-api-access-qv6w4") pod "0899440f-6b0e-465e-a3db-94518b0e7bb5" (UID: "0899440f-6b0e-465e-a3db-94518b0e7bb5"). InnerVolumeSpecName "kube-api-access-qv6w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:09.700831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.700795 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec07bae6-0b99-44c2-a58e-708366cdf86a" (UID: "ec07bae6-0b99-44c2-a58e-708366cdf86a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:40:09.700920 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.700889 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j" (OuterVolumeSpecName: "kube-api-access-22k9j") pod "ec07bae6-0b99-44c2-a58e-708366cdf86a" (UID: "ec07bae6-0b99-44c2-a58e-708366cdf86a"). InnerVolumeSpecName "kube-api-access-22k9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:09.701057 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.700996 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec07bae6-0b99-44c2-a58e-708366cdf86a" (UID: "ec07bae6-0b99-44c2-a58e-708366cdf86a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:40:09.701949 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.701907 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.799319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799284 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22k9j\" (UniqueName: \"kubernetes.io/projected/ec07bae6-0b99-44c2-a58e-708366cdf86a-kube-api-access-22k9j\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.799319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799310 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.799319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799322 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07bae6-0b99-44c2-a58e-708366cdf86a-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.799546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799330 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv6w4\" (UniqueName: \"kubernetes.io/projected/0899440f-6b0e-465e-a3db-94518b0e7bb5-kube-api-access-qv6w4\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.799546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799338 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0899440f-6b0e-465e-a3db-94518b0e7bb5-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.799546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.799347 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07bae6-0b99-44c2-a58e-708366cdf86a-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:40:09.884393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.884358 2571 generic.go:356] "Generic (PLEG): container finished" podID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerID="3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f" exitCode=0 Sep 11 14:40:09.884572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.884422 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" Sep 11 14:40:09.884572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.884446 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" event={"ID":"ec07bae6-0b99-44c2-a58e-708366cdf86a","Type":"ContainerDied","Data":"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f"} Sep 11 14:40:09.884572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.884542 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr" event={"ID":"ec07bae6-0b99-44c2-a58e-708366cdf86a","Type":"ContainerDied","Data":"d8af9cfd682ee6ecaf99d47db3660238cde0f7840e3739b9fa07ff3c01a3bd7f"} Sep 11 14:40:09.884722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.884577 2571 scope.go:117] "RemoveContainer" containerID="3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f" Sep 11 14:40:09.886450 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.886421 2571 generic.go:356] "Generic (PLEG): container finished" podID="0899440f-6b0e-465e-a3db-94518b0e7bb5" containerID="22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979" exitCode=0 Sep 11 14:40:09.886585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.886470 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" Sep 11 14:40:09.886585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.886498 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" event={"ID":"0899440f-6b0e-465e-a3db-94518b0e7bb5","Type":"ContainerDied","Data":"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979"} Sep 11 14:40:09.886585 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.886541 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp" event={"ID":"0899440f-6b0e-465e-a3db-94518b0e7bb5","Type":"ContainerDied","Data":"34cc0a5e750c0bd38c5c6aa29a9a495aa30cfa06adccb91337c94b34f2625d32"} Sep 11 14:40:09.894791 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.894773 2571 scope.go:117] "RemoveContainer" containerID="3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f" Sep 11 14:40:09.895095 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:40:09.895074 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f\": container with ID starting with 3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f not found: ID does not exist" containerID="3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f" Sep 11 14:40:09.895187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.895103 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f"} err="failed to get container status \"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f\": rpc error: code = NotFound desc = could not find container \"3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f\": container with ID starting with 3ce88d9fb28b93c442d798e85b6e4ca7d25f22ec53251fb2552926ea5017f68f not found: ID does not exist" Sep 11 14:40:09.895187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.895122 2571 scope.go:117] "RemoveContainer" containerID="22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979" Sep 11 14:40:09.904946 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.904916 2571 scope.go:117] "RemoveContainer" containerID="22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979" Sep 11 14:40:09.905253 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:40:09.905235 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979\": container with ID starting with 22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979 not found: ID does not exist" containerID="22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979" Sep 11 14:40:09.905315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.905265 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979"} err="failed to get container status \"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979\": rpc error: code = NotFound desc = could not find container \"22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979\": container with ID starting with 22d64eab8dd0a9e3e023c2806dfdd9fa8b7e37797a31e0ee25287995c74a7979 not found: ID does not exist" Sep 11 14:40:09.923250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.923227 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:40:09.929071 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.929049 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b947f70-864d5b69cc-8bxgp"] Sep 11 14:40:09.959333 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.959309 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0899440f-6b0e-465e-a3db-94518b0e7bb5" path="/var/lib/kubelet/pods/0899440f-6b0e-465e-a3db-94518b0e7bb5/volumes" Sep 11 14:40:09.959577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.959564 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:40:09.966428 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:09.966407 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b947f70-99d75b956-6k9hr"] Sep 11 14:40:10.251216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:10.251186 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:11.607382 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:11.607352 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:11.958959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:11.958867 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" path="/var/lib/kubelet/pods/ec07bae6-0b99-44c2-a58e-708366cdf86a/volumes" Sep 11 14:40:12.940309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:12.940279 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:13.479432 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:13.479398 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:14.253736 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:14.253703 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:15.525280 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:15.525248 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:18.550535 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:18.550497 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:28.780469 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:28.780434 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:44.771348 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:44.771301 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:56.785350 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:56.785302 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:57.637686 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:40:57.637649 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:03.102403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:03.102359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:04.747527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:04.747486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:04.860047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:04.860009 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:05.093372 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:05.093319 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:06.631110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:06.631071 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:06.734997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:06.734953 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:07.623076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:07.623032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:07.767517 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:07.767475 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:07.917160 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:07.917070 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:09.449062 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:09.449027 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:16.822130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:16.822095 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:16.931093 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:16.931053 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:17.613780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:17.613741 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:17.718922 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:17.718888 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:24.644398 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:24.644359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:25.617761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:25.617727 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:25.758460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:25.758426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:25.836407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:25.836373 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:25.959229 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:25.959132 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:26.048257 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:26.048210 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:32.861710 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:32.861673 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:36.589844 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:36.589805 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:36.857799 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:36.857701 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:39.364375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:39.364341 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:47.268039 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:47.267992 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:50.003692 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:50.003641 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:57.919149 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:41:57.919106 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:06.866775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:06.866737 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:06.965572 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:06.965533 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:07.375231 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:07.375195 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:07.860772 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:07.860736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:08.124272 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:08.124170 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:08.553913 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:08.553876 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:11.069962 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:11.069913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:11.565700 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:11.565664 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:11.691586 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:11.691550 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:12.345187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:12.345145 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:13.579021 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:13.578980 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:14.638912 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:14.638876 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:15.774556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:15.774512 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:16.811597 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:16.811563 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:16.903880 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:16.903843 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:16.998739 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:16.998701 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:17.775446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.775408 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:17.876058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.876023 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:17.899781 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.899749 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:42:17.900429 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900407 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerName="maas-api" Sep 11 14:42:17.900495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900433 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerName="maas-api" Sep 11 14:42:17.900495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900460 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0899440f-6b0e-465e-a3db-94518b0e7bb5" containerName="payload-processing" Sep 11 14:42:17.900495 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900469 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0899440f-6b0e-465e-a3db-94518b0e7bb5" containerName="payload-processing" Sep 11 14:42:17.900612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900554 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec07bae6-0b99-44c2-a58e-708366cdf86a" containerName="maas-api" Sep 11 14:42:17.900612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.900563 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0899440f-6b0e-465e-a3db-94518b0e7bb5" containerName="payload-processing" Sep 11 14:42:17.904211 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.904190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:17.906504 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.906476 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ef0ff5\"" Sep 11 14:42:17.906633 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.906623 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ef0ff5-dockercfg-89dmr\"" Sep 11 14:42:17.918996 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.918964 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:42:17.967148 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.967115 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:42:17.972569 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.972545 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:17.989820 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:17.989790 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:42:18.004411 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.004372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6nt\" (UniqueName: \"kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.004687 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.004661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ppm\" (UniqueName: \"kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.004864 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.004846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.005121 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.005100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.106131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.106029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.106131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.106114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6nt\" (UniqueName: \"kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.106383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.106175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ppm\" (UniqueName: \"kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.106383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.106250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.107054 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.107030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.107206 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.107079 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.115676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.115631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6nt\" (UniqueName: \"kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt\") pod \"payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.115826 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.115758 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ppm\" (UniqueName: \"kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm\") pod \"payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.215827 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.215791 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:18.296798 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.296751 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:18.495142 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.495115 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:42:18.497678 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:42:18.497652 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24db82e_a2de_4648_b8ae_e353f5089d50.slice/crio-18fb61ffd6051bca1df02dd594fdf5d01936e84067e35c3578cacf9999194489 WatchSource:0}: Error finding container 18fb61ffd6051bca1df02dd594fdf5d01936e84067e35c3578cacf9999194489: Status 404 returned error can't find the container with id 18fb61ffd6051bca1df02dd594fdf5d01936e84067e35c3578cacf9999194489 Sep 11 14:42:18.509916 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.509891 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" event={"ID":"a24db82e-a2de-4648-b8ae-e353f5089d50","Type":"ContainerStarted","Data":"18fb61ffd6051bca1df02dd594fdf5d01936e84067e35c3578cacf9999194489"} Sep 11 14:42:18.578799 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:18.578774 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:42:18.580607 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:42:18.580583 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16950062_f6aa_4b02_a481_3e6efac07b10.slice/crio-473167c33818064475521d07d462fe2a9accea3a40798b92cdb5b08bbac25f75 WatchSource:0}: Error finding container 473167c33818064475521d07d462fe2a9accea3a40798b92cdb5b08bbac25f75: Status 404 returned error can't find the container with id 473167c33818064475521d07d462fe2a9accea3a40798b92cdb5b08bbac25f75 Sep 11 14:42:19.517592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.517560 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" event={"ID":"a24db82e-a2de-4648-b8ae-e353f5089d50","Type":"ContainerStarted","Data":"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6"} Sep 11 14:42:19.518082 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.517670 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:19.519224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.519200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" event={"ID":"16950062-f6aa-4b02-a481-3e6efac07b10","Type":"ContainerStarted","Data":"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2"} Sep 11 14:42:19.519340 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.519240 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" event={"ID":"16950062-f6aa-4b02-a481-3e6efac07b10","Type":"ContainerStarted","Data":"473167c33818064475521d07d462fe2a9accea3a40798b92cdb5b08bbac25f75"} Sep 11 14:42:19.519340 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.519262 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:19.538674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.538623 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" podStartSLOduration=2.538603883 podStartE2EDuration="2.538603883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:19.536486995 +0000 UTC m=+1642.108144479" watchObservedRunningTime="2026-09-11 14:42:19.538603883 +0000 UTC m=+1642.110261355" Sep 11 14:42:19.558437 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:19.558389 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" podStartSLOduration=2.558373135 podStartE2EDuration="2.558373135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:19.556767734 +0000 UTC m=+1642.128425226" watchObservedRunningTime="2026-09-11 14:42:19.558373135 +0000 UTC m=+1642.130030608" Sep 11 14:42:30.526576 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:30.526541 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:42:30.527110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:30.526816 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:42:34.659244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:34.659212 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:39.621345 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:39.621304 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:41.567558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:41.567522 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:41.985970 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:41.985862 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.200455 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:42.200413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.589304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:42.589263 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.783246 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:42.783213 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.885398 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:42.885299 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:43.915904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:43.915875 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:45.201479 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:45.201442 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:45.509275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:45.509168 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:45.607409 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:45.607369 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:46.200442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:46.200375 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:47.672221 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:47.672182 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:48.413267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:48.413228 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:48.523889 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:48.523857 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:48.708452 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:48.708347 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.232048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.232008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.327034 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.326995 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.445309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.445275 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:42:49.449110 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.449091 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.451365 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.451341 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-56b1b6\"" Sep 11 14:42:49.451498 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.451371 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-56b1b6-dockercfg-rncr4\"" Sep 11 14:42:49.461158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.461128 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:42:49.528099 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.528067 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:42:49.531742 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.531724 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.544839 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.544810 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:42:49.611507 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.611468 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.611691 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.611518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7flv\" (UniqueName: \"kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.712232 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.712198 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmfk\" (UniqueName: \"kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.712634 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.712324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.712634 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.712361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.712634 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.712384 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7flv\" (UniqueName: \"kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.713144 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.713128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.722819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.722788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7flv\" (UniqueName: \"kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv\") pod \"payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.760782 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.760743 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:49.813530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.813496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmfk\" (UniqueName: \"kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.813726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.813643 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.814646 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.814618 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.825375 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.825347 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmfk\" (UniqueName: \"kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk\") pod \"payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.847536 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.847254 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:49.887791 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:49.887727 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:50.040919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.040887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:42:50.042878 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:42:50.042848 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee6325b_6c38_45eb_bfc4_ce7ee1268ab3.slice/crio-08f950479ed826253ba1663eee803f54494406d9679dd08ece6b776ce6bf2ab5 WatchSource:0}: Error finding container 08f950479ed826253ba1663eee803f54494406d9679dd08ece6b776ce6bf2ab5: Status 404 returned error can't find the container with id 08f950479ed826253ba1663eee803f54494406d9679dd08ece6b776ce6bf2ab5 Sep 11 14:42:50.138964 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.138916 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:42:50.143233 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:42:50.143203 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc72d01_927e_49d9_a059_17cf092e37ab.slice/crio-b5766b20ad6243180d64303cfb2857806ed55c18073bcd5bb655074932fed009 WatchSource:0}: Error finding container b5766b20ad6243180d64303cfb2857806ed55c18073bcd5bb655074932fed009: Status 404 returned error can't find the container with id b5766b20ad6243180d64303cfb2857806ed55c18073bcd5bb655074932fed009 Sep 11 14:42:50.677721 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.677683 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" event={"ID":"ecc72d01-927e-49d9-a059-17cf092e37ab","Type":"ContainerStarted","Data":"39ef17903013850b6256ccd95acda8f76b7a877655713e248db3edf13708079e"} Sep 11 14:42:50.677957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.677772 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" event={"ID":"ecc72d01-927e-49d9-a059-17cf092e37ab","Type":"ContainerStarted","Data":"b5766b20ad6243180d64303cfb2857806ed55c18073bcd5bb655074932fed009"} Sep 11 14:42:50.677957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.677924 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:42:50.679336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.679310 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" event={"ID":"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3","Type":"ContainerStarted","Data":"e487a7f01ffb98767c2ad833898362f889cdb1d6c8198a68c9668c947dc0973f"} Sep 11 14:42:50.679433 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.679350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" event={"ID":"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3","Type":"ContainerStarted","Data":"08f950479ed826253ba1663eee803f54494406d9679dd08ece6b776ce6bf2ab5"} Sep 11 14:42:50.679490 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.679444 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:42:50.700998 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.700918 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" podStartSLOduration=1.700900029 podStartE2EDuration="1.700900029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:50.698016592 +0000 UTC m=+1673.269674082" watchObservedRunningTime="2026-09-11 14:42:50.700900029 +0000 UTC m=+1673.272557496" Sep 11 14:42:50.719400 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.719336 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" podStartSLOduration=1.719313846 podStartE2EDuration="1.719313846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:50.717780854 +0000 UTC m=+1673.289438352" watchObservedRunningTime="2026-09-11 14:42:50.719313846 +0000 UTC m=+1673.290971322" Sep 11 14:42:50.895986 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:50.895923 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:51.798637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:51.798599 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:52.761188 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:42:52.761139 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:01.686236 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:01.686202 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:43:01.686689 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:01.686395 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:43:10.496315 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:10.496262 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:11.718568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:11.718526 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:12.588340 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:12.588298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:12.654759 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:12.654729 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:13.681074 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:13.680981 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:13.771611 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:13.771580 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:14.252600 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:14.252564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:14.533494 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:14.533457 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:14.633776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:14.633737 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:14.873649 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:14.873537 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:15.603016 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:15.602969 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:15.705376 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:15.705340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:16.521318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:16.521286 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:17.552130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:17.552085 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:17.734905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:17.734869 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:17.827521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:17.827431 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:18.809168 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:18.809125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:19.931159 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:19.931116 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:20.970021 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:20.969979 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:21.985877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:21.985805 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:22.861248 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:22.861210 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:23.727807 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:23.727766 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:24.584294 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:24.584258 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:24.694966 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:24.694925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:24.811351 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:24.811314 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:25.308297 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.308257 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:25.418158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.418119 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:25.536963 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.536913 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:43:25.541020 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.541003 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.543420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.543400 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0ee6c-dockercfg-rsdfj\"" Sep 11 14:43:25.543523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.543402 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0ee6c\"" Sep 11 14:43:25.555079 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.555052 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:43:25.570006 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.569905 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:43:25.573845 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.573801 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.590735 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.590606 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:43:25.665275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.665231 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25tv\" (UniqueName: \"kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.665471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.665283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4n6m\" (UniqueName: \"kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.665471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.665347 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.665471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.665407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.766450 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.766409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25tv\" (UniqueName: \"kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.766662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.766459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4n6m\" (UniqueName: \"kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.766662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.766502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.766662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.766650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.767262 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.767244 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.767433 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.767416 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.778865 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.778838 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25tv\" (UniqueName: \"kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv\") pod \"payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:25.779026 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.778887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4n6m\" (UniqueName: \"kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m\") pod \"payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.852405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.852308 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:25.889013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:25.888976 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:26.058999 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.058926 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:26.149669 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.149529 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:43:26.152071 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:43:26.152042 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418fba0a_a907_4f5c_9ac2_01df87bf0f78.slice/crio-a7bdc04103e4cc901c8120fb8363fb929e35c7026fdabec7cf35e488c9d05f6e WatchSource:0}: Error finding container a7bdc04103e4cc901c8120fb8363fb929e35c7026fdabec7cf35e488c9d05f6e: Status 404 returned error can't find the container with id a7bdc04103e4cc901c8120fb8363fb929e35c7026fdabec7cf35e488c9d05f6e Sep 11 14:43:26.185639 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.185615 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:43:26.187378 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:43:26.187347 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5698bc8_6249_46a9_8b7b_15d7bfc67634.slice/crio-99511de25cfdf09fdb41eb0b512dadbffa2719e5c19ca2f46f5dcb5f0b02d8ec WatchSource:0}: Error finding container 99511de25cfdf09fdb41eb0b512dadbffa2719e5c19ca2f46f5dcb5f0b02d8ec: Status 404 returned error can't find the container with id 99511de25cfdf09fdb41eb0b512dadbffa2719e5c19ca2f46f5dcb5f0b02d8ec Sep 11 14:43:26.866548 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.866508 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" event={"ID":"c5698bc8-6249-46a9-8b7b-15d7bfc67634","Type":"ContainerStarted","Data":"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f"} Sep 11 14:43:26.866973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.866571 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" event={"ID":"c5698bc8-6249-46a9-8b7b-15d7bfc67634","Type":"ContainerStarted","Data":"99511de25cfdf09fdb41eb0b512dadbffa2719e5c19ca2f46f5dcb5f0b02d8ec"} Sep 11 14:43:26.866973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.866603 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:26.868200 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.868174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" event={"ID":"418fba0a-a907-4f5c-9ac2-01df87bf0f78","Type":"ContainerStarted","Data":"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93"} Sep 11 14:43:26.868343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.868215 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" event={"ID":"418fba0a-a907-4f5c-9ac2-01df87bf0f78","Type":"ContainerStarted","Data":"a7bdc04103e4cc901c8120fb8363fb929e35c7026fdabec7cf35e488c9d05f6e"} Sep 11 14:43:26.868343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.868249 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:26.891145 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.891087 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" podStartSLOduration=1.891069224 podStartE2EDuration="1.891069224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:26.889897018 +0000 UTC m=+1709.461554509" watchObservedRunningTime="2026-09-11 14:43:26.891069224 +0000 UTC m=+1709.462726696" Sep 11 14:43:26.910761 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:26.910703 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" podStartSLOduration=1.910684001 podStartE2EDuration="1.910684001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:26.909682811 +0000 UTC m=+1709.481340311" watchObservedRunningTime="2026-09-11 14:43:26.910684001 +0000 UTC m=+1709.482341474" Sep 11 14:43:27.108182 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:27.108146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:28.124970 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:28.124909 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.048109 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.048052 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.169524 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.169489 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:43:34.169766 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.169721 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" podUID="a24db82e-a2de-4648-b8ae-e353f5089d50" containerName="payload-processing" containerID="cri-o://6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6" gracePeriod=30 Sep 11 14:43:34.237320 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.237273 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:43:34.237618 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.237567 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" podUID="16950062-f6aa-4b02-a481-3e6efac07b10" containerName="payload-pre-processing" containerID="cri-o://faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2" gracePeriod=30 Sep 11 14:43:34.256577 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.256496 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.366338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.366283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.541982 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.541948 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:43:34.544669 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.544648 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:43:34.553443 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.553416 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.661368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661272 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ppm\" (UniqueName: \"kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm\") pod \"a24db82e-a2de-4648-b8ae-e353f5089d50\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " Sep 11 14:43:34.661368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661313 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk6nt\" (UniqueName: \"kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt\") pod \"16950062-f6aa-4b02-a481-3e6efac07b10\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " Sep 11 14:43:34.661368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661354 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume\") pod \"a24db82e-a2de-4648-b8ae-e353f5089d50\" (UID: \"a24db82e-a2de-4648-b8ae-e353f5089d50\") " Sep 11 14:43:34.661666 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661431 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume\") pod \"16950062-f6aa-4b02-a481-3e6efac07b10\" (UID: \"16950062-f6aa-4b02-a481-3e6efac07b10\") " Sep 11 14:43:34.661802 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661775 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a24db82e-a2de-4648-b8ae-e353f5089d50" (UID: "a24db82e-a2de-4648-b8ae-e353f5089d50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:43:34.661859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.661817 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16950062-f6aa-4b02-a481-3e6efac07b10" (UID: "16950062-f6aa-4b02-a481-3e6efac07b10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:43:34.663703 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.663671 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm" (OuterVolumeSpecName: "kube-api-access-c7ppm") pod "a24db82e-a2de-4648-b8ae-e353f5089d50" (UID: "a24db82e-a2de-4648-b8ae-e353f5089d50"). InnerVolumeSpecName "kube-api-access-c7ppm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:43:34.663814 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.663716 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt" (OuterVolumeSpecName: "kube-api-access-kk6nt") pod "16950062-f6aa-4b02-a481-3e6efac07b10" (UID: "16950062-f6aa-4b02-a481-3e6efac07b10"). InnerVolumeSpecName "kube-api-access-kk6nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:43:34.670227 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.670185 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:34.762568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.762536 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ppm\" (UniqueName: \"kubernetes.io/projected/a24db82e-a2de-4648-b8ae-e353f5089d50-kube-api-access-c7ppm\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.762568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.762560 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6nt\" (UniqueName: \"kubernetes.io/projected/16950062-f6aa-4b02-a481-3e6efac07b10-kube-api-access-kk6nt\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.762568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.762573 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24db82e-a2de-4648-b8ae-e353f5089d50-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.762811 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.762581 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16950062-f6aa-4b02-a481-3e6efac07b10-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.916098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.916008 2571 generic.go:356] "Generic (PLEG): container finished" podID="a24db82e-a2de-4648-b8ae-e353f5089d50" containerID="6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6" exitCode=0 Sep 11 14:43:34.916098 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.916073 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" Sep 11 14:43:34.916399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.916104 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" event={"ID":"a24db82e-a2de-4648-b8ae-e353f5089d50","Type":"ContainerDied","Data":"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6"} Sep 11 14:43:34.916399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.916171 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj" event={"ID":"a24db82e-a2de-4648-b8ae-e353f5089d50","Type":"ContainerDied","Data":"18fb61ffd6051bca1df02dd594fdf5d01936e84067e35c3578cacf9999194489"} Sep 11 14:43:34.916399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.916201 2571 scope.go:117] "RemoveContainer" containerID="6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6" Sep 11 14:43:34.917651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.917625 2571 generic.go:356] "Generic (PLEG): container finished" podID="16950062-f6aa-4b02-a481-3e6efac07b10" containerID="faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2" exitCode=0 Sep 11 14:43:34.917775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.917678 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" Sep 11 14:43:34.917775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.917708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" event={"ID":"16950062-f6aa-4b02-a481-3e6efac07b10","Type":"ContainerDied","Data":"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2"} Sep 11 14:43:34.917775 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.917755 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv" event={"ID":"16950062-f6aa-4b02-a481-3e6efac07b10","Type":"ContainerDied","Data":"473167c33818064475521d07d462fe2a9accea3a40798b92cdb5b08bbac25f75"} Sep 11 14:43:34.926236 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.926034 2571 scope.go:117] "RemoveContainer" containerID="6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6" Sep 11 14:43:34.926411 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:43:34.926382 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6\": container with ID starting with 6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6 not found: ID does not exist" containerID="6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6" Sep 11 14:43:34.926528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.926416 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6"} err="failed to get container status \"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6\": rpc error: code = NotFound desc = could not find container \"6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6\": container with ID starting with 6431ec391b1a0809e2a3131c52a1ec84f0e74d7f93729354ed66c4d58e468dd6 not found: ID does not exist" Sep 11 14:43:34.926528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.926439 2571 scope.go:117] "RemoveContainer" containerID="faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2" Sep 11 14:43:34.935148 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.935123 2571 scope.go:117] "RemoveContainer" containerID="faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2" Sep 11 14:43:34.935476 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:43:34.935452 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2\": container with ID starting with faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2 not found: ID does not exist" containerID="faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2" Sep 11 14:43:34.935527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.935483 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2"} err="failed to get container status \"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2\": rpc error: code = NotFound desc = could not find container \"faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2\": container with ID starting with faa8acf65887ed2a9f9be6e6a7b7338b2f1fdbebcbc93fc8b552815047916dc2 not found: ID does not exist" Sep 11 14:43:34.952745 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.952717 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:43:34.957950 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.957913 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef0ff5-6746465558-jrwkv"] Sep 11 14:43:34.979405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.979367 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:43:34.984336 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:34.984303 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef0ff5-656d94979f-6vghj"] Sep 11 14:43:35.085224 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:35.085188 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:35.269157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:35.269119 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:35.959752 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:35.959718 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16950062-f6aa-4b02-a481-3e6efac07b10" path="/var/lib/kubelet/pods/16950062-f6aa-4b02-a481-3e6efac07b10/volumes" Sep 11 14:43:35.960150 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:35.960136 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24db82e-a2de-4648-b8ae-e353f5089d50" path="/var/lib/kubelet/pods/a24db82e-a2de-4648-b8ae-e353f5089d50/volumes" Sep 11 14:43:36.247563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:36.247481 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:37.545688 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:37.545652 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:37.877306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:37.877216 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:43:37.877306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:37.877287 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:43:38.571114 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:38.571077 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:39.454919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:39.454885 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:40.568407 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:40.568371 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:41.679179 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:41.679142 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:42.708679 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:42.708637 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:42.893563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:42.893526 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:43.370612 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:43.370574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:43.449788 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:43.449754 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:43.661385 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:43.661298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:43.770698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:43.770659 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:43.996644 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:43.996550 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:44.073747 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:44.073710 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:44.940446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:44.940413 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:46.257676 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:46.257636 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:46.890113 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:46.890073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:47.001022 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:47.000984 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:47.575831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:47.575794 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:48.673971 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:48.673909 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:49.651706 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:49.651663 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:49.814601 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:49.814567 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:49.912695 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:49.912605 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:50.616513 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:50.616477 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:51.612454 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:51.612412 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:52.633183 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:52.633139 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:52.743910 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:52.743867 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:53.080560 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:53.080521 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:53.475293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:53.475192 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:53.575395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:53.575359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:54.000668 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:54.000632 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:54.983528 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:54.983492 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:55.959529 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:43:55.959496 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:01.787792 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.787743 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:01.895562 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.895526 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:44:01.895772 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.895742 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" podUID="ecc72d01-927e-49d9-a059-17cf092e37ab" containerName="payload-processing" containerID="cri-o://39ef17903013850b6256ccd95acda8f76b7a877655713e248db3edf13708079e" gracePeriod=30 Sep 11 14:44:01.930871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.930825 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:44:01.931163 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.931133 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" podUID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" containerName="payload-pre-processing" containerID="cri-o://e487a7f01ffb98767c2ad833898362f889cdb1d6c8198a68c9668c947dc0973f" gracePeriod=30 Sep 11 14:44:01.949586 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:01.949539 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.063045 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.063009 2571 generic.go:356] "Generic (PLEG): container finished" podID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" containerID="e487a7f01ffb98767c2ad833898362f889cdb1d6c8198a68c9668c947dc0973f" exitCode=0 Sep 11 14:44:02.063214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.063082 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" event={"ID":"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3","Type":"ContainerDied","Data":"e487a7f01ffb98767c2ad833898362f889cdb1d6c8198a68c9668c947dc0973f"} Sep 11 14:44:02.064978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.064915 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecc72d01-927e-49d9-a059-17cf092e37ab" containerID="39ef17903013850b6256ccd95acda8f76b7a877655713e248db3edf13708079e" exitCode=0 Sep 11 14:44:02.065136 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.064976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" event={"ID":"ecc72d01-927e-49d9-a059-17cf092e37ab","Type":"ContainerDied","Data":"39ef17903013850b6256ccd95acda8f76b7a877655713e248db3edf13708079e"} Sep 11 14:44:02.066322 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.066293 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.212897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.212862 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.230147 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.230118 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:44:02.233388 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.233363 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:44:02.291726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.291641 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.334783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.334751 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume\") pod \"ecc72d01-927e-49d9-a059-17cf092e37ab\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " Sep 11 14:44:02.335013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.334835 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7flv\" (UniqueName: \"kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv\") pod \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " Sep 11 14:44:02.335013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.334858 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhmfk\" (UniqueName: \"kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk\") pod \"ecc72d01-927e-49d9-a059-17cf092e37ab\" (UID: \"ecc72d01-927e-49d9-a059-17cf092e37ab\") " Sep 11 14:44:02.335013 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.334904 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume\") pod \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\" (UID: \"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3\") " Sep 11 14:44:02.335331 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.335305 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecc72d01-927e-49d9-a059-17cf092e37ab" (UID: "ecc72d01-927e-49d9-a059-17cf092e37ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:02.335432 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.335372 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" (UID: "7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:02.337088 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.337065 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv" (OuterVolumeSpecName: "kube-api-access-j7flv") pod "7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" (UID: "7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3"). InnerVolumeSpecName "kube-api-access-j7flv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:02.337195 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.337179 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk" (OuterVolumeSpecName: "kube-api-access-qhmfk") pod "ecc72d01-927e-49d9-a059-17cf092e37ab" (UID: "ecc72d01-927e-49d9-a059-17cf092e37ab"). InnerVolumeSpecName "kube-api-access-qhmfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:02.436403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.436368 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc72d01-927e-49d9-a059-17cf092e37ab-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.436403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.436396 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7flv\" (UniqueName: \"kubernetes.io/projected/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-kube-api-access-j7flv\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.436403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.436406 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmfk\" (UniqueName: \"kubernetes.io/projected/ecc72d01-927e-49d9-a059-17cf092e37ab-kube-api-access-qhmfk\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.436403 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.436415 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.677380 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.677285 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.778637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:02.778601 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:03.077928 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.077901 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" Sep 11 14:44:03.078418 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.077923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld" event={"ID":"7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3","Type":"ContainerDied","Data":"08f950479ed826253ba1663eee803f54494406d9679dd08ece6b776ce6bf2ab5"} Sep 11 14:44:03.078418 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.078051 2571 scope.go:117] "RemoveContainer" containerID="e487a7f01ffb98767c2ad833898362f889cdb1d6c8198a68c9668c947dc0973f" Sep 11 14:44:03.079432 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.079410 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" event={"ID":"ecc72d01-927e-49d9-a059-17cf092e37ab","Type":"ContainerDied","Data":"b5766b20ad6243180d64303cfb2857806ed55c18073bcd5bb655074932fed009"} Sep 11 14:44:03.079514 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.079433 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw" Sep 11 14:44:03.088116 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.088082 2571 scope.go:117] "RemoveContainer" containerID="39ef17903013850b6256ccd95acda8f76b7a877655713e248db3edf13708079e" Sep 11 14:44:03.106822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.106796 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:44:03.116564 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.116543 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56b1b6-785b97f978-hvcld"] Sep 11 14:44:03.132409 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.132386 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:44:03.138683 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.138663 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-56b1b6-8648669f7c-42jjw"] Sep 11 14:44:03.810530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.810492 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:03.959405 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.959371 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" path="/var/lib/kubelet/pods/7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3/volumes" Sep 11 14:44:03.959749 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:03.959736 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc72d01-927e-49d9-a059-17cf092e37ab" path="/var/lib/kubelet/pods/ecc72d01-927e-49d9-a059-17cf092e37ab/volumes" Sep 11 14:44:05.091164 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:05.091113 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:06.327189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:06.327146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:07.502267 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:07.502219 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:08.723538 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:08.723503 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:10.022662 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:10.022611 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:11.364616 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:11.364574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:11.464905 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:11.464870 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:11.683633 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:11.683545 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:11.982250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:11.982168 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:12.833625 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:12.833592 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:14.003418 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:14.003380 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:15.351050 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:15.351013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:16.791726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:16.791684 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:17.944954 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:17.944913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:18.907697 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:18.907661 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:19.819450 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:19.819417 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:19.919354 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:19.919319 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:22.368637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:22.368593 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.570609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.570566 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.629836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.629800 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:44:23.630132 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.630092 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" podUID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" containerName="payload-processing" containerID="cri-o://16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885" gracePeriod=30 Sep 11 14:44:23.645120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.645089 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:44:23.645359 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.645336 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" podUID="e386cbc9-5494-4464-be94-83b163194f6d" containerName="payload-pre-processing" containerID="cri-o://11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3" gracePeriod=30 Sep 11 14:44:23.679735 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.679694 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.761349 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.761315 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.935794 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.935697 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.938506 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.938483 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:44:23.952750 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:23.952721 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:44:24.045332 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045298 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume\") pod \"e386cbc9-5494-4464-be94-83b163194f6d\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " Sep 11 14:44:24.045530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045352 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fppg\" (UniqueName: \"kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg\") pod \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " Sep 11 14:44:24.045530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045432 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf6qz\" (UniqueName: \"kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz\") pod \"e386cbc9-5494-4464-be94-83b163194f6d\" (UID: \"e386cbc9-5494-4464-be94-83b163194f6d\") " Sep 11 14:44:24.045530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045499 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume\") pod \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\" (UID: \"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8\") " Sep 11 14:44:24.045783 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045756 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e386cbc9-5494-4464-be94-83b163194f6d" (UID: "e386cbc9-5494-4464-be94-83b163194f6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:24.045842 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.045780 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" (UID: "2d0bb9bb-90cc-4b78-880f-cf736a65b8c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:24.047633 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.047603 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg" (OuterVolumeSpecName: "kube-api-access-7fppg") pod "2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" (UID: "2d0bb9bb-90cc-4b78-880f-cf736a65b8c8"). InnerVolumeSpecName "kube-api-access-7fppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:24.047822 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.047804 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz" (OuterVolumeSpecName: "kube-api-access-kf6qz") pod "e386cbc9-5494-4464-be94-83b163194f6d" (UID: "e386cbc9-5494-4464-be94-83b163194f6d"). InnerVolumeSpecName "kube-api-access-kf6qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:24.051170 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.051139 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:24.147135 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.147042 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6qz\" (UniqueName: \"kubernetes.io/projected/e386cbc9-5494-4464-be94-83b163194f6d-kube-api-access-kf6qz\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.147135 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.147075 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.147135 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.147085 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386cbc9-5494-4464-be94-83b163194f6d-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.147135 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.147095 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fppg\" (UniqueName: \"kubernetes.io/projected/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8-kube-api-access-7fppg\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.190274 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.190235 2571 generic.go:356] "Generic (PLEG): container finished" podID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" containerID="16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885" exitCode=0 Sep 11 14:44:24.190449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.190281 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" Sep 11 14:44:24.190449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.190318 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" event={"ID":"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8","Type":"ContainerDied","Data":"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885"} Sep 11 14:44:24.190449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.190369 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h" event={"ID":"2d0bb9bb-90cc-4b78-880f-cf736a65b8c8","Type":"ContainerDied","Data":"393eaf6415406caca0c5574d1c03e24c753acf07c1bc637d0b1ff3e66390d988"} Sep 11 14:44:24.190449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.190393 2571 scope.go:117] "RemoveContainer" containerID="16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885" Sep 11 14:44:24.191977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.191926 2571 generic.go:356] "Generic (PLEG): container finished" podID="e386cbc9-5494-4464-be94-83b163194f6d" containerID="11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3" exitCode=0 Sep 11 14:44:24.192105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.192000 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" event={"ID":"e386cbc9-5494-4464-be94-83b163194f6d","Type":"ContainerDied","Data":"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3"} Sep 11 14:44:24.192105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.192014 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" Sep 11 14:44:24.192105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.192054 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp" event={"ID":"e386cbc9-5494-4464-be94-83b163194f6d","Type":"ContainerDied","Data":"d31e6b90165995f9ec2e3a4a841d2857485c00be67cf807868924454a08eac70"} Sep 11 14:44:24.199671 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.199647 2571 scope.go:117] "RemoveContainer" containerID="16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885" Sep 11 14:44:24.200180 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:44:24.200159 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885\": container with ID starting with 16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885 not found: ID does not exist" containerID="16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885" Sep 11 14:44:24.200244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.200191 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885"} err="failed to get container status \"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885\": rpc error: code = NotFound desc = could not find container \"16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885\": container with ID starting with 16fa19684e3f662b153b2c9cd686b14d2d1e361932fbb6dccefdd7d9b8f03885 not found: ID does not exist" Sep 11 14:44:24.200244 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.200209 2571 scope.go:117] "RemoveContainer" containerID="11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3" Sep 11 14:44:24.209016 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.208997 2571 scope.go:117] "RemoveContainer" containerID="11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3" Sep 11 14:44:24.209298 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:44:24.209279 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3\": container with ID starting with 11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3 not found: ID does not exist" containerID="11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3" Sep 11 14:44:24.209338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.209306 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3"} err="failed to get container status \"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3\": rpc error: code = NotFound desc = could not find container \"11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3\": container with ID starting with 11af3dd631dfa92f4bc0d91c19b1a17a90c69908baad7eb8dcab80a7bf4275a3 not found: ID does not exist" Sep 11 14:44:24.220844 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.220812 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:44:24.233318 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.233286 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-958398-84d8c9d746-wbbbp"] Sep 11 14:44:24.252977 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.252923 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:44:24.256106 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.256079 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-958398-77b677489c-zwc7h"] Sep 11 14:44:24.405333 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.405236 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:24.503463 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:24.503423 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:25.835180 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:25.835144 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:25.959959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:25.959912 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" path="/var/lib/kubelet/pods/2d0bb9bb-90cc-4b78-880f-cf736a65b8c8/volumes" Sep 11 14:44:25.960311 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:25.960297 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e386cbc9-5494-4464-be94-83b163194f6d" path="/var/lib/kubelet/pods/e386cbc9-5494-4464-be94-83b163194f6d/volumes" Sep 11 14:44:27.106342 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:27.106305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:28.105792 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:28.105754 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:29.171522 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:29.171485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:30.128295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:30.128262 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:31.632301 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:31.632251 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:31.734765 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:31.734733 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:31.908995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:31.908879 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:33.491654 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:33.491614 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:34.415573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:34.415527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:40.541907 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:40.541864 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:41.607567 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:41.607533 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:42.537253 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:42.537205 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.387751 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.387718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.434708 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.434670 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:44:43.435006 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.434966 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" podUID="a3d9c982-9d32-438b-98b6-4227d39d838e" containerName="payload-pre-processing" containerID="cri-o://541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7" gracePeriod=30 Sep 11 14:44:43.484324 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.484280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.573997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.573948 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.669975 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.669873 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.707712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.707686 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:44:43.747096 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.747060 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:43.839246 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.839212 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2fzr\" (UniqueName: \"kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr\") pod \"a3d9c982-9d32-438b-98b6-4227d39d838e\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " Sep 11 14:44:43.839421 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.839302 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume\") pod \"a3d9c982-9d32-438b-98b6-4227d39d838e\" (UID: \"a3d9c982-9d32-438b-98b6-4227d39d838e\") " Sep 11 14:44:43.839640 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.839615 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3d9c982-9d32-438b-98b6-4227d39d838e" (UID: "a3d9c982-9d32-438b-98b6-4227d39d838e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:43.841358 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.841327 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr" (OuterVolumeSpecName: "kube-api-access-m2fzr") pod "a3d9c982-9d32-438b-98b6-4227d39d838e" (UID: "a3d9c982-9d32-438b-98b6-4227d39d838e"). InnerVolumeSpecName "kube-api-access-m2fzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:43.941023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.940914 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fzr\" (UniqueName: \"kubernetes.io/projected/a3d9c982-9d32-438b-98b6-4227d39d838e-kube-api-access-m2fzr\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:43.941023 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:43.940971 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d9c982-9d32-438b-98b6-4227d39d838e-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:44:44.111539 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.111504 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:44.299058 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.299024 2571 generic.go:356] "Generic (PLEG): container finished" podID="a3d9c982-9d32-438b-98b6-4227d39d838e" containerID="541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7" exitCode=0 Sep 11 14:44:44.299266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.299078 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" Sep 11 14:44:44.299266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.299109 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" event={"ID":"a3d9c982-9d32-438b-98b6-4227d39d838e","Type":"ContainerDied","Data":"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7"} Sep 11 14:44:44.299266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.299165 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s" event={"ID":"a3d9c982-9d32-438b-98b6-4227d39d838e","Type":"ContainerDied","Data":"07c8ad184137dabf09226f1f00d286759290e2dd9f7d419da37038a981746a4c"} Sep 11 14:44:44.299266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.299186 2571 scope.go:117] "RemoveContainer" containerID="541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7" Sep 11 14:44:44.308644 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.308625 2571 scope.go:117] "RemoveContainer" containerID="541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7" Sep 11 14:44:44.308988 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:44:44.308969 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7\": container with ID starting with 541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7 not found: ID does not exist" containerID="541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7" Sep 11 14:44:44.309068 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.309000 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7"} err="failed to get container status \"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7\": rpc error: code = NotFound desc = could not find container \"541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7\": container with ID starting with 541237104f568739a4249e22ea3a7db33076687c5429edecb7330c225883daf7 not found: ID does not exist" Sep 11 14:44:44.316276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.316247 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:44:44.323223 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.323195 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-455f2e-7b54bd9566-hkq4s"] Sep 11 14:44:44.932420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:44.932386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:45.961271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:45.961233 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d9c982-9d32-438b-98b6-4227d39d838e" path="/var/lib/kubelet/pods/a3d9c982-9d32-438b-98b6-4227d39d838e/volumes" Sep 11 14:44:46.109063 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:46.109013 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:47.252175 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:47.252144 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:48.351343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:48.351289 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:51.301990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:51.301886 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:51.384496 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:51.384457 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:51.601163 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:51.601069 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:56.022782 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:56.022749 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x_c5698bc8-6249-46a9-8b7b-15d7bfc67634/payload-processing/0.log" Sep 11 14:44:56.598250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:56.598215 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:56.968393 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:56.968301 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:57.185356 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:57.185316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:58.027725 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:58.027698 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:44:58.034699 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:58.034671 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:44:59.565026 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:44:59.564989 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:00.070338 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.070298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:00.194008 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.193975 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs"] Sep 11 14:45:00.194374 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194362 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16950062-f6aa-4b02-a481-3e6efac07b10" containerName="payload-pre-processing" Sep 11 14:45:00.194419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194376 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="16950062-f6aa-4b02-a481-3e6efac07b10" containerName="payload-pre-processing" Sep 11 14:45:00.194419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194390 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d9c982-9d32-438b-98b6-4227d39d838e" containerName="payload-pre-processing" Sep 11 14:45:00.194419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194396 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d9c982-9d32-438b-98b6-4227d39d838e" containerName="payload-pre-processing" Sep 11 14:45:00.194419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194403 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24db82e-a2de-4648-b8ae-e353f5089d50" containerName="payload-processing" Sep 11 14:45:00.194419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194409 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24db82e-a2de-4648-b8ae-e353f5089d50" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194421 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194427 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194435 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194441 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194454 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc72d01-927e-49d9-a059-17cf092e37ab" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194459 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc72d01-927e-49d9-a059-17cf092e37ab" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194465 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e386cbc9-5494-4464-be94-83b163194f6d" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194470 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e386cbc9-5494-4464-be94-83b163194f6d" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194525 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d9c982-9d32-438b-98b6-4227d39d838e" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194533 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e386cbc9-5494-4464-be94-83b163194f6d" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194539 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="16950062-f6aa-4b02-a481-3e6efac07b10" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194544 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee6325b-6c38-45eb-bfc4-ce7ee1268ab3" containerName="payload-pre-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194549 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc72d01-927e-49d9-a059-17cf092e37ab" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194554 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24db82e-a2de-4648-b8ae-e353f5089d50" containerName="payload-processing" Sep 11 14:45:00.194573 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.194562 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0bb9bb-90cc-4b78-880f-cf736a65b8c8" containerName="payload-processing" Sep 11 14:45:00.197590 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.197562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" Sep 11 14:45:00.204518 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.204385 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\"" Sep 11 14:45:00.206086 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.206052 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:00.208328 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.208302 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" Sep 11 14:45:00.224204 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.224175 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs"] Sep 11 14:45:00.506271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.506238 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs"] Sep 11 14:45:00.507888 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:45:00.507862 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0cf145_a616_4426_8436_d2f87e52aa85.slice/crio-85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d WatchSource:0}: Error finding container 85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d: Status 404 returned error can't find the container with id 85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d Sep 11 14:45:00.510036 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:00.510011 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:45:01.080973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:01.080926 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:01.383729 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:01.383633 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" event={"ID":"2b0cf145-a616-4426-8436-d2f87e52aa85","Type":"ContainerStarted","Data":"85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d"} Sep 11 14:45:02.540863 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:02.540821 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:03.395886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:03.395849 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" event={"ID":"2b0cf145-a616-4426-8436-d2f87e52aa85","Type":"ContainerStarted","Data":"a7482d0636a9fefeca876f61baa573544b33a3576f4e0513c1ca2c6cd77a2573"} Sep 11 14:45:03.424710 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:03.424651 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" podStartSLOduration=0.940424482 podStartE2EDuration="3.424635317s" totalImagesPullingTime="2.484210835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:03.421913389 +0000 UTC m=+1805.993570881" watchObservedRunningTime="2026-09-11 14:45:03.424635317 +0000 UTC m=+1805.996292790" Sep 11 14:45:03.733271 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:03.733184 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:04.402588 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:04.402558 2571 generic.go:356] "Generic (PLEG): container finished" podID="2b0cf145-a616-4426-8436-d2f87e52aa85" containerID="a7482d0636a9fefeca876f61baa573544b33a3576f4e0513c1ca2c6cd77a2573" exitCode=0 Sep 11 14:45:04.402780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:04.402651 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" event={"ID":"2b0cf145-a616-4426-8436-d2f87e52aa85","Type":"ContainerDied","Data":"a7482d0636a9fefeca876f61baa573544b33a3576f4e0513c1ca2c6cd77a2573"} Sep 11 14:45:05.144556 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:05.144518 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:05.540335 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:05.540308 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" Sep 11 14:45:06.409972 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:06.409906 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:06.420460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:06.419762 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" event={"ID":"2b0cf145-a616-4426-8436-d2f87e52aa85","Type":"ContainerDied","Data":"85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d"} Sep 11 14:45:06.420460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:06.419868 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85853232d836380f619ba4de873d009e92df498d91271a46e1c768e78a3e516d" Sep 11 14:45:06.420460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:06.419980 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs" Sep 11 14:45:06.519067 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:06.519034 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.093304 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.093265 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.257087 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.257043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.360655 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.360556 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.457583 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.457549 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:07.457995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.457980 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0cf145-a616-4426-8436-d2f87e52aa85" containerName="cleanup" Sep 11 14:45:07.458039 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.457997 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0cf145-a616-4426-8436-d2f87e52aa85" containerName="cleanup" Sep 11 14:45:07.458092 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.458063 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0cf145-a616-4426-8436-d2f87e52aa85" containerName="cleanup" Sep 11 14:45:07.461621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.461604 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.464607 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.464588 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8f7967-dockercfg-ppq44\"" Sep 11 14:45:07.466056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.466041 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8f7967\"" Sep 11 14:45:07.485446 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.485419 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:07.519925 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.519894 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:07.523704 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.523684 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.540491 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.540465 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:07.568105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.568073 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcl5z\" (UniqueName: \"kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.568264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.568115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.568264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.568172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.568264 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.568196 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hch8d\" (UniqueName: \"kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.668965 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.668861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.668965 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.668902 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hch8d\" (UniqueName: \"kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.669204 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.669101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcl5z\" (UniqueName: \"kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.669204 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.669141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.669747 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.669722 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.669877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.669859 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.686458 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.686429 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hch8d\" (UniqueName: \"kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d\") pod \"payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:07.686810 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.686790 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcl5z\" (UniqueName: \"kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.771764 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.771724 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:07.834273 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:07.834245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:08.047837 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.047812 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:08.049843 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:45:08.049814 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4500db0_9932_41e2_8d2c_ce36bf7af37b.slice/crio-415770c35c8864a25bcd5ccd4e3bff4c136bc73af757da0a2d4bfa705d0084bf WatchSource:0}: Error finding container 415770c35c8864a25bcd5ccd4e3bff4c136bc73af757da0a2d4bfa705d0084bf: Status 404 returned error can't find the container with id 415770c35c8864a25bcd5ccd4e3bff4c136bc73af757da0a2d4bfa705d0084bf Sep 11 14:45:08.118252 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.118224 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:08.120587 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:45:08.120562 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38fe696_0731_420f_8446_dfcc645219ed.slice/crio-5a7f5090c433e093f7ef1d769308e06b1eca06535fbcc95e16a7dd8597c80bf6 WatchSource:0}: Error finding container 5a7f5090c433e093f7ef1d769308e06b1eca06535fbcc95e16a7dd8597c80bf6: Status 404 returned error can't find the container with id 5a7f5090c433e093f7ef1d769308e06b1eca06535fbcc95e16a7dd8597c80bf6 Sep 11 14:45:08.431449 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.431341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" event={"ID":"d4500db0-9932-41e2-8d2c-ce36bf7af37b","Type":"ContainerStarted","Data":"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9"} Sep 11 14:45:08.431651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.431557 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:08.431651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.431606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" event={"ID":"d4500db0-9932-41e2-8d2c-ce36bf7af37b","Type":"ContainerStarted","Data":"415770c35c8864a25bcd5ccd4e3bff4c136bc73af757da0a2d4bfa705d0084bf"} Sep 11 14:45:08.433059 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.433028 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" event={"ID":"c38fe696-0731-420f-8446-dfcc645219ed","Type":"ContainerStarted","Data":"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194"} Sep 11 14:45:08.433196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.433083 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" event={"ID":"c38fe696-0731-420f-8446-dfcc645219ed","Type":"ContainerStarted","Data":"5a7f5090c433e093f7ef1d769308e06b1eca06535fbcc95e16a7dd8597c80bf6"} Sep 11 14:45:08.433196 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.433111 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:08.457682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.457634 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" podStartSLOduration=1.45761827 podStartE2EDuration="1.45761827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:08.455238941 +0000 UTC m=+1811.026896414" watchObservedRunningTime="2026-09-11 14:45:08.45761827 +0000 UTC m=+1811.029275789" Sep 11 14:45:08.480780 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:08.480730 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" podStartSLOduration=1.480713261 podStartE2EDuration="1.480713261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:08.47949044 +0000 UTC m=+1811.051147933" watchObservedRunningTime="2026-09-11 14:45:08.480713261 +0000 UTC m=+1811.052370733" Sep 11 14:45:19.439722 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:19.439688 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:19.440169 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:19.439778 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:19.679527 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:19.679462 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:19.767383 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:19.767283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:30.016659 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:30.016608 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:30.299886 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:30.299856 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:31.581790 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:31.581749 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:33.051419 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:33.051385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:34.455740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:34.455699 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:35.581694 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:35.581644 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:36.857105 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:36.846420 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:38.135468 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:38.135426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.931804 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:41.931770 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:42.973660 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:42.973625 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.070042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.070008 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:43.070272 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.070244 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" podUID="c38fe696-0731-420f-8446-dfcc645219ed" containerName="payload-processing" containerID="cri-o://dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194" gracePeriod=30 Sep 11 14:45:43.082536 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.082502 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:43.082774 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.082746 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" podUID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" containerName="payload-pre-processing" containerID="cri-o://c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9" gracePeriod=30 Sep 11 14:45:43.159949 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.159897 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.303257 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.303190 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.335423 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.335397 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:43.359188 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.359163 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:43.417222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.416682 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume\") pod \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " Sep 11 14:45:43.417222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.416778 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcl5z\" (UniqueName: \"kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z\") pod \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\" (UID: \"d4500db0-9932-41e2-8d2c-ce36bf7af37b\") " Sep 11 14:45:43.417222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.417140 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hch8d\" (UniqueName: \"kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d\") pod \"c38fe696-0731-420f-8446-dfcc645219ed\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " Sep 11 14:45:43.417222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.417219 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4500db0-9932-41e2-8d2c-ce36bf7af37b" (UID: "d4500db0-9932-41e2-8d2c-ce36bf7af37b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:43.418547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.417573 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume\") pod \"c38fe696-0731-420f-8446-dfcc645219ed\" (UID: \"c38fe696-0731-420f-8446-dfcc645219ed\") " Sep 11 14:45:43.418547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.418003 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c38fe696-0731-420f-8446-dfcc645219ed" (UID: "c38fe696-0731-420f-8446-dfcc645219ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:43.418547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.418351 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38fe696-0731-420f-8446-dfcc645219ed-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.418547 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.418372 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4500db0-9932-41e2-8d2c-ce36bf7af37b-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.421172 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.421141 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d" (OuterVolumeSpecName: "kube-api-access-hch8d") pod "c38fe696-0731-420f-8446-dfcc645219ed" (UID: "c38fe696-0731-420f-8446-dfcc645219ed"). InnerVolumeSpecName "kube-api-access-hch8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:43.422341 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.422317 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z" (OuterVolumeSpecName: "kube-api-access-mcl5z") pod "d4500db0-9932-41e2-8d2c-ce36bf7af37b" (UID: "d4500db0-9932-41e2-8d2c-ce36bf7af37b"). InnerVolumeSpecName "kube-api-access-mcl5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:43.485225 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.485188 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.519276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.519192 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcl5z\" (UniqueName: \"kubernetes.io/projected/d4500db0-9932-41e2-8d2c-ce36bf7af37b-kube-api-access-mcl5z\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.519276 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.519226 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hch8d\" (UniqueName: \"kubernetes.io/projected/c38fe696-0731-420f-8446-dfcc645219ed-kube-api-access-hch8d\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.613316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.613282 2571 generic.go:356] "Generic (PLEG): container finished" podID="c38fe696-0731-420f-8446-dfcc645219ed" containerID="dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194" exitCode=0 Sep 11 14:45:43.613520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.613351 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" Sep 11 14:45:43.613520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.613366 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" event={"ID":"c38fe696-0731-420f-8446-dfcc645219ed","Type":"ContainerDied","Data":"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194"} Sep 11 14:45:43.613520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.613452 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf" event={"ID":"c38fe696-0731-420f-8446-dfcc645219ed","Type":"ContainerDied","Data":"5a7f5090c433e093f7ef1d769308e06b1eca06535fbcc95e16a7dd8597c80bf6"} Sep 11 14:45:43.613520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.613483 2571 scope.go:117] "RemoveContainer" containerID="dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194" Sep 11 14:45:43.615073 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.615047 2571 generic.go:356] "Generic (PLEG): container finished" podID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" containerID="c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9" exitCode=0 Sep 11 14:45:43.615187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.615112 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" Sep 11 14:45:43.615187 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.615147 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" event={"ID":"d4500db0-9932-41e2-8d2c-ce36bf7af37b","Type":"ContainerDied","Data":"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9"} Sep 11 14:45:43.615316 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.615200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6" event={"ID":"d4500db0-9932-41e2-8d2c-ce36bf7af37b","Type":"ContainerDied","Data":"415770c35c8864a25bcd5ccd4e3bff4c136bc73af757da0a2d4bfa705d0084bf"} Sep 11 14:45:43.624150 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.624052 2571 scope.go:117] "RemoveContainer" containerID="dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194" Sep 11 14:45:43.624381 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:45:43.624360 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194\": container with ID starting with dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194 not found: ID does not exist" containerID="dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194" Sep 11 14:45:43.624473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.624389 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194"} err="failed to get container status \"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194\": rpc error: code = NotFound desc = could not find container \"dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194\": container with ID starting with dc93e5b32cedb8d0288b6fbdc79101629a9faf6885c6f47e3ebdeb85da258194 not found: ID does not exist" Sep 11 14:45:43.624473 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.624407 2571 scope.go:117] "RemoveContainer" containerID="c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9" Sep 11 14:45:43.632924 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.632908 2571 scope.go:117] "RemoveContainer" containerID="c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9" Sep 11 14:45:43.633212 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:45:43.633194 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9\": container with ID starting with c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9 not found: ID does not exist" containerID="c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9" Sep 11 14:45:43.633266 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.633219 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9"} err="failed to get container status \"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9\": rpc error: code = NotFound desc = could not find container \"c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9\": container with ID starting with c417d260d6f3d9e498cf78fd72d2da47d4f2e566428644c2d49b97dd6f8bcfe9 not found: ID does not exist" Sep 11 14:45:43.644509 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.644489 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:43.652162 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.652136 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f7967-f94d6d6cd-xgrx6"] Sep 11 14:45:43.652334 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.652312 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.666486 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.666316 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:43.669368 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.669347 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f7967-5fc64b95bf-sklqf"] Sep 11 14:45:43.959861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.959827 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38fe696-0731-420f-8446-dfcc645219ed" path="/var/lib/kubelet/pods/c38fe696-0731-420f-8446-dfcc645219ed/volumes" Sep 11 14:45:43.960238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.960224 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" path="/var/lib/kubelet/pods/d4500db0-9932-41e2-8d2c-ce36bf7af37b/volumes" Sep 11 14:45:43.984189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:43.984157 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:44.092072 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:44.092043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:45.313439 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:45.313407 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:46.560442 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:46.560409 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:47.931740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:47.931703 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:49.119038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:49.118998 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:51.882717 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:51.882675 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:51.986250 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:51.986214 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:52.268183 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:52.268106 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:56.081273 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:56.081238 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:56.182459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:56.182429 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:56.737465 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:56.737430 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:57.964725 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:57.964689 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:59.113176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:45:59.113136 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:00.048575 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:00.048546 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:01.193880 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:01.193848 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:02.292829 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:02.292789 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:12.164521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.164488 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:12.202621 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.202590 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:46:12.202841 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.202806 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" podUID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" containerName="payload-processing" containerID="cri-o://a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f" gracePeriod=30 Sep 11 14:46:12.214871 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.214838 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:46:12.215138 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.215109 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" podUID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" containerName="payload-pre-processing" containerID="cri-o://c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93" gracePeriod=30 Sep 11 14:46:12.503512 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.503481 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:46:12.506205 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.506185 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:46:12.598230 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598192 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l25tv\" (UniqueName: \"kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv\") pod \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " Sep 11 14:46:12.598430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598259 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4n6m\" (UniqueName: \"kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m\") pod \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " Sep 11 14:46:12.598430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598375 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume\") pod \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\" (UID: \"418fba0a-a907-4f5c-9ac2-01df87bf0f78\") " Sep 11 14:46:12.598430 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598395 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume\") pod \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\" (UID: \"c5698bc8-6249-46a9-8b7b-15d7bfc67634\") " Sep 11 14:46:12.598801 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598768 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "418fba0a-a907-4f5c-9ac2-01df87bf0f78" (UID: "418fba0a-a907-4f5c-9ac2-01df87bf0f78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:46:12.598952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.598826 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5698bc8-6249-46a9-8b7b-15d7bfc67634" (UID: "c5698bc8-6249-46a9-8b7b-15d7bfc67634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:46:12.600855 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.600836 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv" (OuterVolumeSpecName: "kube-api-access-l25tv") pod "c5698bc8-6249-46a9-8b7b-15d7bfc67634" (UID: "c5698bc8-6249-46a9-8b7b-15d7bfc67634"). InnerVolumeSpecName "kube-api-access-l25tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:46:12.600997 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.600862 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m" (OuterVolumeSpecName: "kube-api-access-f4n6m") pod "418fba0a-a907-4f5c-9ac2-01df87bf0f78" (UID: "418fba0a-a907-4f5c-9ac2-01df87bf0f78"). InnerVolumeSpecName "kube-api-access-f4n6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:46:12.700222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.700136 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418fba0a-a907-4f5c-9ac2-01df87bf0f78-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.700222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.700166 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5698bc8-6249-46a9-8b7b-15d7bfc67634-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.700222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.700175 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l25tv\" (UniqueName: \"kubernetes.io/projected/c5698bc8-6249-46a9-8b7b-15d7bfc67634-kube-api-access-l25tv\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.700222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.700185 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4n6m\" (UniqueName: \"kubernetes.io/projected/418fba0a-a907-4f5c-9ac2-01df87bf0f78-kube-api-access-f4n6m\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.766575 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.766537 2571 generic.go:356] "Generic (PLEG): container finished" podID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" containerID="c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93" exitCode=0 Sep 11 14:46:12.766755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.766585 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" Sep 11 14:46:12.766755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.766621 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" event={"ID":"418fba0a-a907-4f5c-9ac2-01df87bf0f78","Type":"ContainerDied","Data":"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93"} Sep 11 14:46:12.766755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.766677 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk" event={"ID":"418fba0a-a907-4f5c-9ac2-01df87bf0f78","Type":"ContainerDied","Data":"a7bdc04103e4cc901c8120fb8363fb929e35c7026fdabec7cf35e488c9d05f6e"} Sep 11 14:46:12.766755 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.766703 2571 scope.go:117] "RemoveContainer" containerID="c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93" Sep 11 14:46:12.768202 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.768180 2571 generic.go:356] "Generic (PLEG): container finished" podID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" containerID="a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f" exitCode=0 Sep 11 14:46:12.768323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.768224 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" Sep 11 14:46:12.768323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.768235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" event={"ID":"c5698bc8-6249-46a9-8b7b-15d7bfc67634","Type":"ContainerDied","Data":"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f"} Sep 11 14:46:12.768323 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.768274 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x" event={"ID":"c5698bc8-6249-46a9-8b7b-15d7bfc67634","Type":"ContainerDied","Data":"99511de25cfdf09fdb41eb0b512dadbffa2719e5c19ca2f46f5dcb5f0b02d8ec"} Sep 11 14:46:12.776295 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.776168 2571 scope.go:117] "RemoveContainer" containerID="c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93" Sep 11 14:46:12.776503 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:46:12.776476 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93\": container with ID starting with c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93 not found: ID does not exist" containerID="c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93" Sep 11 14:46:12.776581 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.776514 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93"} err="failed to get container status \"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93\": rpc error: code = NotFound desc = could not find container \"c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93\": container with ID starting with c11990c2feea0d2b9e25bf417ee8f548ce32800d3b18f4c506e5d7ec77380f93 not found: ID does not exist" Sep 11 14:46:12.776581 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.776537 2571 scope.go:117] "RemoveContainer" containerID="a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f" Sep 11 14:46:12.785176 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.785159 2571 scope.go:117] "RemoveContainer" containerID="a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f" Sep 11 14:46:12.785458 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:46:12.785441 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f\": container with ID starting with a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f not found: ID does not exist" containerID="a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f" Sep 11 14:46:12.785523 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.785463 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f"} err="failed to get container status \"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f\": rpc error: code = NotFound desc = could not find container \"a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f\": container with ID starting with a717daaca52cf96fc102c700db08589072ed83fb32233724446be5e020be524f not found: ID does not exist" Sep 11 14:46:12.793743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.793722 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:46:12.798395 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.798369 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0ee6c-5d8768576c-5qz7x"] Sep 11 14:46:12.816566 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.816539 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:46:12.821592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:12.821573 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0ee6c-8bf74cf86-89wjk"] Sep 11 14:46:13.578309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:13.578264 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:13.793050 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:13.793010 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:13.958563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:13.958485 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" path="/var/lib/kubelet/pods/418fba0a-a907-4f5c-9ac2-01df87bf0f78/volumes" Sep 11 14:46:13.958864 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:13.958851 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" path="/var/lib/kubelet/pods/c5698bc8-6249-46a9-8b7b-15d7bfc67634/volumes" Sep 11 14:46:14.915804 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:14.915768 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:15.909737 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:15.909708 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:17.920581 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:17.920538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:24.269978 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:24.269868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:51.261048 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:51.260976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:54.865460 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:54.865419 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:56.124259 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:46:56.124224 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:02.963041 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:02.963002 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:03.049848 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:03.049809 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:03.286603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:03.286569 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:03.914420 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:03.914368 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:04.944635 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:04.944597 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:05.890531 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:05.890489 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:06.907088 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:06.907046 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:07.828483 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:07.828442 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:10.031433 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:10.031395 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:13.995854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:13.995810 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:14.730602 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:14.730560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:14.731700 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:14.731679 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs"] Sep 11 14:47:14.740648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:14.740624 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-40d3a2-29818965-w9pfs"] Sep 11 14:47:14.848108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:14.848064 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:14.946438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:14.946404 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:15.047127 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:15.047096 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:15.138579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:15.138540 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:15.481042 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:15.480956 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:15.609055 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:15.609020 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:15.958995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:15.958954 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b0cf145-a616-4426-8436-d2f87e52aa85" path="/var/lib/kubelet/pods/2b0cf145-a616-4426-8436-d2f87e52aa85/volumes" Sep 11 14:47:16.715877 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:16.715833 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:17.918289 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:17.918254 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:22.566076 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:22.566041 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:22.665854 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:22.665822 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:22.896849 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:22.896761 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:23.495806 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:23.495767 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:24.435753 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:24.435708 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:25.613952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:25.613907 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:26.538693 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:26.538652 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.080028 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.079988 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:47:34.080603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.080187 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" podUID="36764d3d-19d0-470e-8eb9-640832ffd114" containerName="maas-api" containerID="cri-o://30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc" gracePeriod=30 Sep 11 14:47:34.116263 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.116227 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.166705 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.166654 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:47:34.167558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.167500 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" containerName="payload-pre-processing" containerID="cri-o://41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4" gracePeriod=30 Sep 11 14:47:34.194682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.194652 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.293069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.293032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.418343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.418310 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.425891 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.425865 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:47:34.447216 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.447175 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:47:34.543475 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.543439 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.562471 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562446 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7xqz\" (UniqueName: \"kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz\") pod \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " Sep 11 14:47:34.562609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562481 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") pod \"36764d3d-19d0-470e-8eb9-640832ffd114\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " Sep 11 14:47:34.562609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562517 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca\") pod \"36764d3d-19d0-470e-8eb9-640832ffd114\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " Sep 11 14:47:34.562609 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562581 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") pod \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\" (UID: \"c9f3b930-07dd-47a0-bc4c-37896d0c305a\") " Sep 11 14:47:34.562760 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562654 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs\") pod \"36764d3d-19d0-470e-8eb9-640832ffd114\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " Sep 11 14:47:34.562760 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562679 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl9s4\" (UniqueName: \"kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4\") pod \"36764d3d-19d0-470e-8eb9-640832ffd114\" (UID: \"36764d3d-19d0-470e-8eb9-640832ffd114\") " Sep 11 14:47:34.563092 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.562916 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36764d3d-19d0-470e-8eb9-640832ffd114" (UID: "36764d3d-19d0-470e-8eb9-640832ffd114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:34.563092 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.563035 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9f3b930-07dd-47a0-bc4c-37896d0c305a" (UID: "c9f3b930-07dd-47a0-bc4c-37896d0c305a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:34.563275 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.563104 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36764d3d-19d0-470e-8eb9-640832ffd114-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.564893 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.564862 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36764d3d-19d0-470e-8eb9-640832ffd114" (UID: "36764d3d-19d0-470e-8eb9-640832ffd114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:34.565027 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.564899 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz" (OuterVolumeSpecName: "kube-api-access-c7xqz") pod "c9f3b930-07dd-47a0-bc4c-37896d0c305a" (UID: "c9f3b930-07dd-47a0-bc4c-37896d0c305a"). InnerVolumeSpecName "kube-api-access-c7xqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:34.565222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.565194 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36764d3d-19d0-470e-8eb9-640832ffd114" (UID: "36764d3d-19d0-470e-8eb9-640832ffd114"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:34.565222 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.565205 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4" (OuterVolumeSpecName: "kube-api-access-cl9s4") pod "36764d3d-19d0-470e-8eb9-640832ffd114" (UID: "36764d3d-19d0-470e-8eb9-640832ffd114"). InnerVolumeSpecName "kube-api-access-cl9s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:34.663904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.663797 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9f3b930-07dd-47a0-bc4c-37896d0c305a-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.663904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.663834 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.663904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.663847 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9s4\" (UniqueName: \"kubernetes.io/projected/36764d3d-19d0-470e-8eb9-640832ffd114-kube-api-access-cl9s4\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.663904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.663859 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xqz\" (UniqueName: \"kubernetes.io/projected/c9f3b930-07dd-47a0-bc4c-37896d0c305a-kube-api-access-c7xqz\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.663904 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.663873 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36764d3d-19d0-470e-8eb9-640832ffd114-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:34.819530 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.819490 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:34.921188 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:34.921088 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:35.171955 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.171847 2571 generic.go:356] "Generic (PLEG): container finished" podID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" containerID="41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4" exitCode=0 Sep 11 14:47:35.171955 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.171899 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" Sep 11 14:47:35.171955 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.171942 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" event={"ID":"c9f3b930-07dd-47a0-bc4c-37896d0c305a","Type":"ContainerDied","Data":"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4"} Sep 11 14:47:35.172598 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.171993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2" event={"ID":"c9f3b930-07dd-47a0-bc4c-37896d0c305a","Type":"ContainerDied","Data":"e3bc39c7eb6519970f5a613ec357cce216f57bedf77cff9679348d17afe0ea0e"} Sep 11 14:47:35.172598 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.172016 2571 scope.go:117] "RemoveContainer" containerID="41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4" Sep 11 14:47:35.173459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.173375 2571 generic.go:356] "Generic (PLEG): container finished" podID="36764d3d-19d0-470e-8eb9-640832ffd114" containerID="30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc" exitCode=0 Sep 11 14:47:35.173459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.173398 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" event={"ID":"36764d3d-19d0-470e-8eb9-640832ffd114","Type":"ContainerDied","Data":"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc"} Sep 11 14:47:35.173459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.173419 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" Sep 11 14:47:35.173459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.173444 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r" event={"ID":"36764d3d-19d0-470e-8eb9-640832ffd114","Type":"ContainerDied","Data":"aa60ee6812a3da7211a9236045c7d3250e46e89afb9a3ff8628b5dc5bebdc62d"} Sep 11 14:47:35.181333 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.181199 2571 scope.go:117] "RemoveContainer" containerID="41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4" Sep 11 14:47:35.181637 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:47:35.181609 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4\": container with ID starting with 41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4 not found: ID does not exist" containerID="41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4" Sep 11 14:47:35.181733 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.181649 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4"} err="failed to get container status \"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4\": rpc error: code = NotFound desc = could not find container \"41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4\": container with ID starting with 41d743eff4d5f84bdcebb79bc308f213e376fffb7dfaf24e3088de178f3939d4 not found: ID does not exist" Sep 11 14:47:35.181733 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.181668 2571 scope.go:117] "RemoveContainer" containerID="30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc" Sep 11 14:47:35.190639 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.190621 2571 scope.go:117] "RemoveContainer" containerID="30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc" Sep 11 14:47:35.191011 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:47:35.190990 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc\": container with ID starting with 30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc not found: ID does not exist" containerID="30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc" Sep 11 14:47:35.191069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.191020 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc"} err="failed to get container status \"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc\": rpc error: code = NotFound desc = could not find container \"30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc\": container with ID starting with 30edc2855481f8260670a75aff19be3257fb707116e127ed836e39cccfa9e8fc not found: ID does not exist" Sep 11 14:47:35.197818 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.197791 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:47:35.205173 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.205145 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-09b16c-69d9c59b5c-8st6r"] Sep 11 14:47:35.218238 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.218207 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:47:35.227399 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.227359 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-09b16c-588f646c7d-6dfk2"] Sep 11 14:47:35.760620 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.760583 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:35.959731 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.959699 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36764d3d-19d0-470e-8eb9-640832ffd114" path="/var/lib/kubelet/pods/36764d3d-19d0-470e-8eb9-640832ffd114/volumes" Sep 11 14:47:35.960114 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:35.960099 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" path="/var/lib/kubelet/pods/c9f3b930-07dd-47a0-bc4c-37896d0c305a/volumes" Sep 11 14:47:41.633185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:41.633151 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:41.733229 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:41.733194 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:41.966223 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:41.966125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:43.453744 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:43.453704 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:44.501660 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:44.501613 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.279660 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.279563 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.324890 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.324859 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:47:53.325283 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.325138 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" podUID="8f49c792-6069-4ea0-bd8b-e783111ae550" containerName="payload-processing" containerID="cri-o://98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace" gracePeriod=30 Sep 11 14:47:53.332270 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.332198 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:47:53.332497 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.332439 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" podUID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" containerName="payload-pre-processing" containerID="cri-o://ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c" gracePeriod=30 Sep 11 14:47:53.377210 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.377172 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.482128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.482082 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.572991 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.572963 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:47:53.601925 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.601898 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.606884 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.606863 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:47:53.639953 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.639912 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj9zw\" (UniqueName: \"kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw\") pod \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " Sep 11 14:47:53.640119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.640016 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume\") pod \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\" (UID: \"3ffd3cb9-bf29-4d80-93d4-e281986451b6\") " Sep 11 14:47:53.640119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.640038 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume\") pod \"8f49c792-6069-4ea0-bd8b-e783111ae550\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " Sep 11 14:47:53.640119 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.640095 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvs7f\" (UniqueName: \"kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f\") pod \"8f49c792-6069-4ea0-bd8b-e783111ae550\" (UID: \"8f49c792-6069-4ea0-bd8b-e783111ae550\") " Sep 11 14:47:53.640437 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.640411 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f49c792-6069-4ea0-bd8b-e783111ae550" (UID: "8f49c792-6069-4ea0-bd8b-e783111ae550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:53.640437 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.640423 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ffd3cb9-bf29-4d80-93d4-e281986451b6" (UID: "3ffd3cb9-bf29-4d80-93d4-e281986451b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:53.642012 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.641987 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw" (OuterVolumeSpecName: "kube-api-access-nj9zw") pod "3ffd3cb9-bf29-4d80-93d4-e281986451b6" (UID: "3ffd3cb9-bf29-4d80-93d4-e281986451b6"). InnerVolumeSpecName "kube-api-access-nj9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:53.642199 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.642174 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f" (OuterVolumeSpecName: "kube-api-access-cvs7f") pod "8f49c792-6069-4ea0-bd8b-e783111ae550" (UID: "8f49c792-6069-4ea0-bd8b-e783111ae550"). InnerVolumeSpecName "kube-api-access-cvs7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:53.707019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.706974 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:53.741157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.741127 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9zw\" (UniqueName: \"kubernetes.io/projected/3ffd3cb9-bf29-4d80-93d4-e281986451b6-kube-api-access-nj9zw\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.741157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.741150 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffd3cb9-bf29-4d80-93d4-e281986451b6-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.741157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.741161 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f49c792-6069-4ea0-bd8b-e783111ae550-plugins-config-volume\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.741408 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:53.741169 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvs7f\" (UniqueName: \"kubernetes.io/projected/8f49c792-6069-4ea0-bd8b-e783111ae550-kube-api-access-cvs7f\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:47:54.083379 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.083344 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:54.206653 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.206609 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:54.266565 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.266477 2571 generic.go:356] "Generic (PLEG): container finished" podID="8f49c792-6069-4ea0-bd8b-e783111ae550" containerID="98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace" exitCode=0 Sep 11 14:47:54.266565 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.266516 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" Sep 11 14:47:54.266784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.266559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" event={"ID":"8f49c792-6069-4ea0-bd8b-e783111ae550","Type":"ContainerDied","Data":"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace"} Sep 11 14:47:54.266784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.266610 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf" event={"ID":"8f49c792-6069-4ea0-bd8b-e783111ae550","Type":"ContainerDied","Data":"1f03fd96c3dee83a252cbec84e4f972f8bd1d8386443f071b6673fb86b3a3029"} Sep 11 14:47:54.266784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.266641 2571 scope.go:117] "RemoveContainer" containerID="98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace" Sep 11 14:47:54.267966 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.267945 2571 generic.go:356] "Generic (PLEG): container finished" podID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" containerID="ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c" exitCode=0 Sep 11 14:47:54.268069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.267977 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" event={"ID":"3ffd3cb9-bf29-4d80-93d4-e281986451b6","Type":"ContainerDied","Data":"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c"} Sep 11 14:47:54.268069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.268035 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" event={"ID":"3ffd3cb9-bf29-4d80-93d4-e281986451b6","Type":"ContainerDied","Data":"c91f882f4bf9640bb0de35fc8d46655f73127be972eaebdb71279ef573312370"} Sep 11 14:47:54.268069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.267986 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k" Sep 11 14:47:54.275679 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.275654 2571 scope.go:117] "RemoveContainer" containerID="98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace" Sep 11 14:47:54.276041 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:47:54.275927 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace\": container with ID starting with 98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace not found: ID does not exist" containerID="98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace" Sep 11 14:47:54.276104 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.276050 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace"} err="failed to get container status \"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace\": rpc error: code = NotFound desc = could not find container \"98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace\": container with ID starting with 98ae33c9b2ba8c5a98764cdda4b575bb944daafaac9d2f3c203ef8fe73f06ace not found: ID does not exist" Sep 11 14:47:54.276104 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.276066 2571 scope.go:117] "RemoveContainer" containerID="ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c" Sep 11 14:47:54.283636 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.283615 2571 scope.go:117] "RemoveContainer" containerID="ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c" Sep 11 14:47:54.283922 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:47:54.283899 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c\": container with ID starting with ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c not found: ID does not exist" containerID="ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c" Sep 11 14:47:54.283998 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.283920 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c"} err="failed to get container status \"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c\": rpc error: code = NotFound desc = could not find container \"ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c\": container with ID starting with ac13ff5378f418d52851462e8105a2bee89a085386abaf0a3524872c621e781c not found: ID does not exist" Sep 11 14:47:54.289427 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.289406 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:47:54.294674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.294653 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1c284e-b8b5bc4b6-wzc6k"] Sep 11 14:47:54.313259 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.313237 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:47:54.314779 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:54.314757 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1c284e-75f8f7b86c-w4nxf"] Sep 11 14:47:55.958494 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:55.958456 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" path="/var/lib/kubelet/pods/3ffd3cb9-bf29-4d80-93d4-e281986451b6/volumes" Sep 11 14:47:55.958972 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:47:55.958955 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f49c792-6069-4ea0-bd8b-e783111ae550" path="/var/lib/kubelet/pods/8f49c792-6069-4ea0-bd8b-e783111ae550/volumes" Sep 11 14:48:00.976859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:00.976824 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:01.049571 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:01.049543 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:01.302867 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:01.302835 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:02.741664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:02.741625 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:02.840198 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:02.840167 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:04.173956 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.173910 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:48:04.174463 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.174185 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77989fd775-lknmd" podUID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" containerName="manager" containerID="cri-o://0f36e5254c3126304352f0c1ad077251a6b097cd586f2749cbb193d7a3741cc4" gracePeriod=30 Sep 11 14:48:04.320410 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.320373 2571 generic.go:356] "Generic (PLEG): container finished" podID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" containerID="0f36e5254c3126304352f0c1ad077251a6b097cd586f2749cbb193d7a3741cc4" exitCode=0 Sep 11 14:48:04.320579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.320448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-lknmd" event={"ID":"e8fab5aa-9618-46bf-baa1-4885dedf45b9","Type":"ContainerDied","Data":"0f36e5254c3126304352f0c1ad077251a6b097cd586f2749cbb193d7a3741cc4"} Sep 11 14:48:04.419128 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.419100 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:48:04.543648 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.543617 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpf2x\" (UniqueName: \"kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x\") pod \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " Sep 11 14:48:04.543852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.543730 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs\") pod \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " Sep 11 14:48:04.543852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.543756 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert\") pod \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\" (UID: \"e8fab5aa-9618-46bf-baa1-4885dedf45b9\") " Sep 11 14:48:04.545906 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.545877 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e8fab5aa-9618-46bf-baa1-4885dedf45b9" (UID: "e8fab5aa-9618-46bf-baa1-4885dedf45b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:48:04.546049 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.545924 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8fab5aa-9618-46bf-baa1-4885dedf45b9" (UID: "e8fab5aa-9618-46bf-baa1-4885dedf45b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:48:04.546129 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.546043 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x" (OuterVolumeSpecName: "kube-api-access-wpf2x") pod "e8fab5aa-9618-46bf-baa1-4885dedf45b9" (UID: "e8fab5aa-9618-46bf-baa1-4885dedf45b9"). InnerVolumeSpecName "kube-api-access-wpf2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:48:04.644784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.644744 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpf2x\" (UniqueName: \"kubernetes.io/projected/e8fab5aa-9618-46bf-baa1-4885dedf45b9-kube-api-access-wpf2x\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:48:04.644784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.644774 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:48:04.644784 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:04.644783 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8fab5aa-9618-46bf-baa1-4885dedf45b9-webhook-cert\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:48:05.327012 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.326975 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-lknmd" Sep 11 14:48:05.327637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.327016 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-lknmd" event={"ID":"e8fab5aa-9618-46bf-baa1-4885dedf45b9","Type":"ContainerDied","Data":"44d834dbc0cc528c23fc31499889ca345b4922c669992e371602fa7dabd326d6"} Sep 11 14:48:05.327637 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.327082 2571 scope.go:117] "RemoveContainer" containerID="0f36e5254c3126304352f0c1ad077251a6b097cd586f2749cbb193d7a3741cc4" Sep 11 14:48:05.350270 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.350247 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:48:05.364353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.364329 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77989fd775-lknmd"] Sep 11 14:48:05.963836 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:05.963803 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" path="/var/lib/kubelet/pods/e8fab5aa-9618-46bf-baa1-4885dedf45b9/volumes" Sep 11 14:48:07.702740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:07.702709 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:07.788157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:07.788130 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:28.878190 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:48:28.878159 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:06.282562 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282525 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282896 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38fe696-0731-420f-8446-dfcc645219ed" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282907 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38fe696-0731-420f-8446-dfcc645219ed" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282921 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282927 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282952 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282961 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282971 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282976 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282990 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36764d3d-19d0-470e-8eb9-640832ffd114" containerName="maas-api" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.282997 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="36764d3d-19d0-470e-8eb9-640832ffd114" containerName="maas-api" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283004 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283009 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283016 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f49c792-6069-4ea0-bd8b-e783111ae550" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283021 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f49c792-6069-4ea0-bd8b-e783111ae550" containerName="payload-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283031 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283037 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" containerName="payload-pre-processing" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283045 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" containerName="manager" Sep 11 14:49:06.283070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283050 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" containerName="manager" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283107 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5698bc8-6249-46a9-8b7b-15d7bfc67634" containerName="payload-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283115 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f3b930-07dd-47a0-bc4c-37896d0c305a" containerName="payload-pre-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283121 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38fe696-0731-420f-8446-dfcc645219ed" containerName="payload-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283126 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f49c792-6069-4ea0-bd8b-e783111ae550" containerName="payload-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283131 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fab5aa-9618-46bf-baa1-4885dedf45b9" containerName="manager" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283136 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffd3cb9-bf29-4d80-93d4-e281986451b6" containerName="payload-pre-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283141 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4500db0-9932-41e2-8d2c-ce36bf7af37b" containerName="payload-pre-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283147 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="418fba0a-a907-4f5c-9ac2-01df87bf0f78" containerName="payload-pre-processing" Sep 11 14:49:06.283690 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.283152 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="36764d3d-19d0-470e-8eb9-640832ffd114" containerName="maas-api" Sep 11 14:49:06.286367 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.286346 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.290740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.290710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 14:49:06.290740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.290726 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 14:49:06.290740 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.290737 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jhxc4\"" Sep 11 14:49:06.291056 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.290712 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 14:49:06.303293 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.303263 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:06.402655 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.402621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.402831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.402677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwh8\" (UniqueName: \"kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.402831 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.402814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.504033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.503993 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.504033 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.504040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.504265 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.504096 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwh8\" (UniqueName: \"kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.506571 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.506546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.506697 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.506586 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.513957 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.513920 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwh8\" (UniqueName: \"kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8\") pod \"maas-controller-77989fd775-gv764\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.597121 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.597040 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:06.872319 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:06.872292 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:06.874975 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:49:06.874944 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e0c333_2429_4be2_94af_d8023e5ff5de.slice/crio-2e4a4cd8578e85b6464c7e1755072130b890df9dbc31e5dc9fe87a809195928b WatchSource:0}: Error finding container 2e4a4cd8578e85b6464c7e1755072130b890df9dbc31e5dc9fe87a809195928b: Status 404 returned error can't find the container with id 2e4a4cd8578e85b6464c7e1755072130b890df9dbc31e5dc9fe87a809195928b Sep 11 14:49:07.615512 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:07.615480 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-gv764" event={"ID":"72e0c333-2429-4be2-94af-d8023e5ff5de","Type":"ContainerStarted","Data":"2e4a4cd8578e85b6464c7e1755072130b890df9dbc31e5dc9fe87a809195928b"} Sep 11 14:49:08.621891 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:08.621851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-gv764" event={"ID":"72e0c333-2429-4be2-94af-d8023e5ff5de","Type":"ContainerStarted","Data":"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2"} Sep 11 14:49:08.622369 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:08.621955 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:08.643158 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:08.643102 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77989fd775-gv764" podStartSLOduration=1.937093435 podStartE2EDuration="2.64307664s" totalImagesPullingTime="705.983205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:08.642983124 +0000 UTC m=+2051.214640617" watchObservedRunningTime="2026-09-11 14:49:08.64307664 +0000 UTC m=+2051.214734113" Sep 11 14:49:19.628445 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:19.628406 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:29.068951 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.068907 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:29.176520 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.176487 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:29.176730 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.176691 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77989fd775-gv764" podUID="72e0c333-2429-4be2-94af-d8023e5ff5de" containerName="manager" containerID="cri-o://0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2" gracePeriod=30 Sep 11 14:49:29.422385 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.422352 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:29.506326 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.506299 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs\") pod \"72e0c333-2429-4be2-94af-d8023e5ff5de\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " Sep 11 14:49:29.506498 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.506332 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert\") pod \"72e0c333-2429-4be2-94af-d8023e5ff5de\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " Sep 11 14:49:29.506498 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.506357 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfwh8\" (UniqueName: \"kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8\") pod \"72e0c333-2429-4be2-94af-d8023e5ff5de\" (UID: \"72e0c333-2429-4be2-94af-d8023e5ff5de\") " Sep 11 14:49:29.508645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.508604 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8" (OuterVolumeSpecName: "kube-api-access-mfwh8") pod "72e0c333-2429-4be2-94af-d8023e5ff5de" (UID: "72e0c333-2429-4be2-94af-d8023e5ff5de"). InnerVolumeSpecName "kube-api-access-mfwh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:49:29.508777 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.508665 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "72e0c333-2429-4be2-94af-d8023e5ff5de" (UID: "72e0c333-2429-4be2-94af-d8023e5ff5de"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:49:29.509080 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.509062 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72e0c333-2429-4be2-94af-d8023e5ff5de" (UID: "72e0c333-2429-4be2-94af-d8023e5ff5de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:49:29.607592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.607507 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:49:29.607592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.607545 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e0c333-2429-4be2-94af-d8023e5ff5de-webhook-cert\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:49:29.607592 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.607558 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwh8\" (UniqueName: \"kubernetes.io/projected/72e0c333-2429-4be2-94af-d8023e5ff5de-kube-api-access-mfwh8\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:49:29.719792 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.719757 2571 generic.go:356] "Generic (PLEG): container finished" podID="72e0c333-2429-4be2-94af-d8023e5ff5de" containerID="0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2" exitCode=0 Sep 11 14:49:29.719981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.719801 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-gv764" Sep 11 14:49:29.719981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.719842 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-gv764" event={"ID":"72e0c333-2429-4be2-94af-d8023e5ff5de","Type":"ContainerDied","Data":"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2"} Sep 11 14:49:29.719981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.719892 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-gv764" event={"ID":"72e0c333-2429-4be2-94af-d8023e5ff5de","Type":"ContainerDied","Data":"2e4a4cd8578e85b6464c7e1755072130b890df9dbc31e5dc9fe87a809195928b"} Sep 11 14:49:29.719981 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.719913 2571 scope.go:117] "RemoveContainer" containerID="0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2" Sep 11 14:49:29.729484 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.729468 2571 scope.go:117] "RemoveContainer" containerID="0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2" Sep 11 14:49:29.729718 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:49:29.729699 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2\": container with ID starting with 0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2 not found: ID does not exist" containerID="0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2" Sep 11 14:49:29.729763 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.729727 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2"} err="failed to get container status \"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2\": rpc error: code = NotFound desc = could not find container \"0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2\": container with ID starting with 0846cc2dd813ebfb69bc11f761c3bc52ab9a5867adbfe574e3aefcf613d5f9a2 not found: ID does not exist" Sep 11 14:49:29.743438 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.743417 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:29.751990 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.751969 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77989fd775-gv764"] Sep 11 14:49:29.958131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:29.958050 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e0c333-2429-4be2-94af-d8023e5ff5de" path="/var/lib/kubelet/pods/72e0c333-2429-4be2-94af-d8023e5ff5de/volumes" Sep 11 14:49:30.551070 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.551029 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:30.694166 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.694124 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77989fd775-bkppn"] Sep 11 14:49:30.694590 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.694572 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e0c333-2429-4be2-94af-d8023e5ff5de" containerName="manager" Sep 11 14:49:30.694682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.694593 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e0c333-2429-4be2-94af-d8023e5ff5de" containerName="manager" Sep 11 14:49:30.694682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.694671 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e0c333-2429-4be2-94af-d8023e5ff5de" containerName="manager" Sep 11 14:49:30.699093 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.699071 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.701743 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.701724 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 14:49:30.702002 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.701980 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 14:49:30.702120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.701982 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jhxc4\"" Sep 11 14:49:30.702120 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.701982 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 14:49:30.711038 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.711018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-bkppn"] Sep 11 14:49:30.824108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.824012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-webhook-cert\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.824108 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.824063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2bw\" (UniqueName: \"kubernetes.io/projected/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-kube-api-access-sw2bw\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.824299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.824161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-metrics-certs\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.924881 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.924839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-webhook-cert\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.925089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.924888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2bw\" (UniqueName: \"kubernetes.io/projected/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-kube-api-access-sw2bw\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.925089 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.924973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-metrics-certs\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.927568 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.927537 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-metrics-certs\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.927684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.927657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-webhook-cert\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:30.934558 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:30.934530 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2bw\" (UniqueName: \"kubernetes.io/projected/9d590040-f0e5-4e43-9b46-7ebeaa6d15b4-kube-api-access-sw2bw\") pod \"maas-controller-77989fd775-bkppn\" (UID: \"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4\") " pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:31.009165 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:31.009120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:31.286299 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:31.286273 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77989fd775-bkppn"] Sep 11 14:49:31.288358 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:49:31.288330 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d590040_f0e5_4e43_9b46_7ebeaa6d15b4.slice/crio-bfac420c84f6e53733f16db7e4733498f4e00faa80aef3449b617a2c7b937fae WatchSource:0}: Error finding container bfac420c84f6e53733f16db7e4733498f4e00faa80aef3449b617a2c7b937fae: Status 404 returned error can't find the container with id bfac420c84f6e53733f16db7e4733498f4e00faa80aef3449b617a2c7b937fae Sep 11 14:49:31.731314 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:31.731285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-bkppn" event={"ID":"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4","Type":"ContainerStarted","Data":"bfac420c84f6e53733f16db7e4733498f4e00faa80aef3449b617a2c7b937fae"} Sep 11 14:49:32.738510 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:32.738474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77989fd775-bkppn" event={"ID":"9d590040-f0e5-4e43-9b46-7ebeaa6d15b4","Type":"ContainerStarted","Data":"7544206f385ea4636bc31c10c25ff2982b32c0e22964a85f3e95938e1960ab11"} Sep 11 14:49:32.738998 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:32.738616 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:32.776015 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:32.775958 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77989fd775-bkppn" podStartSLOduration=2.382405764 podStartE2EDuration="2.775930362s" totalImagesPullingTime="393.524598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:32.774726599 +0000 UTC m=+2075.346384072" watchObservedRunningTime="2026-09-11 14:49:32.775930362 +0000 UTC m=+2075.347587870" Sep 11 14:49:43.746204 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:43.746172 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77989fd775-bkppn" Sep 11 14:49:58.056429 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:58.056401 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:49:58.064922 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:49:58.064902 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:50:56.924643 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:50:56.924557 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:56.995231 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:50:56.995192 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:51:58.255270 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:51:58.255236 2571 scope.go:117] "RemoveContainer" containerID="a7482d0636a9fefeca876f61baa573544b33a3576f4e0513c1ca2c6cd77a2573" Sep 11 14:52:31.017584 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:52:31.017513 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:43.626567 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:52:43.626535 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:45.200302 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:52:45.200261 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:45.303651 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:52:45.303615 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:53:09.350282 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:09.350243 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:53:11.526376 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:11.526338 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:53:38.100645 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.100608 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r"] Sep 11 14:53:38.104016 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.103994 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.125828 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.125800 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r"] Sep 11 14:53:38.181819 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.181784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175354e-fc94-415a-ba41-22273bb42031-openshift-service-ca\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.182006 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.181833 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-maas-api-tls\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.182069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.182006 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nn2\" (UniqueName: \"kubernetes.io/projected/4175354e-fc94-415a-ba41-22273bb42031-kube-api-access-t2nn2\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.182069 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.182043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-metrics-certs\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.283223 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.283186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-maas-api-tls\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.283396 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.283303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nn2\" (UniqueName: \"kubernetes.io/projected/4175354e-fc94-415a-ba41-22273bb42031-kube-api-access-t2nn2\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.283396 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.283329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-metrics-certs\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.283396 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.283346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175354e-fc94-415a-ba41-22273bb42031-openshift-service-ca\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.284053 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.284030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4175354e-fc94-415a-ba41-22273bb42031-openshift-service-ca\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.285681 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.285659 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-maas-api-tls\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.285762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.285699 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4175354e-fc94-415a-ba41-22273bb42031-metrics-certs\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.293579 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.293559 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nn2\" (UniqueName: \"kubernetes.io/projected/4175354e-fc94-415a-ba41-22273bb42031-kube-api-access-t2nn2\") pod \"maas-api-54b97d6fdf-ch47r\" (UID: \"4175354e-fc94-415a-ba41-22273bb42031\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.414209 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.414138 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:38.689633 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.689609 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r"] Sep 11 14:53:38.691834 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:53:38.691800 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4175354e_fc94_415a_ba41_22273bb42031.slice/crio-eb3deaeb13a0a3331e09d8a19fdd532ff915518f9e157c7629c1fab1a1bae2ed WatchSource:0}: Error finding container eb3deaeb13a0a3331e09d8a19fdd532ff915518f9e157c7629c1fab1a1bae2ed: Status 404 returned error can't find the container with id eb3deaeb13a0a3331e09d8a19fdd532ff915518f9e157c7629c1fab1a1bae2ed Sep 11 14:53:38.693189 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.693166 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:53:38.856354 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:38.856323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" event={"ID":"4175354e-fc94-415a-ba41-22273bb42031","Type":"ContainerStarted","Data":"eb3deaeb13a0a3331e09d8a19fdd532ff915518f9e157c7629c1fab1a1bae2ed"} Sep 11 14:53:39.862600 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:39.862559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" event={"ID":"4175354e-fc94-415a-ba41-22273bb42031","Type":"ContainerStarted","Data":"b5e40e3b2d261dcc13e035a1dace53d2f8a99bd9bb336868adea0ebeb0862133"} Sep 11 14:53:39.863072 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:39.862703 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:39.906995 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:39.906910 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" podStartSLOduration=1.555117372 podStartE2EDuration="1.906894029s" totalImagesPullingTime="351.776657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:39.906315958 +0000 UTC m=+2322.477973456" watchObservedRunningTime="2026-09-11 14:53:39.906894029 +0000 UTC m=+2322.478551522" Sep 11 14:53:45.872737 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:45.872707 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-ch47r" Sep 11 14:53:45.938901 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:45.938866 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:53:45.939117 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:45.939093 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" podUID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" containerName="maas-api" containerID="cri-o://8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4" gracePeriod=30 Sep 11 14:53:47.188719 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.188692 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:53:47.370482 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.370443 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca\") pod \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " Sep 11 14:53:47.370682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.370550 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs\") pod \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " Sep 11 14:53:47.370682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.370576 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls\") pod \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " Sep 11 14:53:47.370682 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.370632 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwc69\" (UniqueName: \"kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69\") pod \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\" (UID: \"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989\") " Sep 11 14:53:47.370847 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.370806 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" (UID: "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:53:47.371019 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.371001 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-openshift-service-ca\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:53:47.372792 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.372767 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" (UID: "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:53:47.372898 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.372805 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" (UID: "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:53:47.372959 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.372898 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69" (OuterVolumeSpecName: "kube-api-access-xwc69") pod "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" (UID: "f7be5fdc-0e9c-46c5-80c1-0dd9c279c989"). InnerVolumeSpecName "kube-api-access-xwc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:53:47.471776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.471745 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-metrics-certs\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:53:47.471776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.471769 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-maas-api-tls\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:53:47.471776 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.471778 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwc69\" (UniqueName: \"kubernetes.io/projected/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989-kube-api-access-xwc69\") on node \"ip-10-0-141-198.ec2.internal\" DevicePath \"\"" Sep 11 14:53:47.901478 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.901445 2571 generic.go:356] "Generic (PLEG): container finished" podID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" containerID="8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4" exitCode=0 Sep 11 14:53:47.901664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.901511 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" Sep 11 14:53:47.901664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.901529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" event={"ID":"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989","Type":"ContainerDied","Data":"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4"} Sep 11 14:53:47.901664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.901592 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-rkdch" event={"ID":"f7be5fdc-0e9c-46c5-80c1-0dd9c279c989","Type":"ContainerDied","Data":"aaf900ecb02d16b35d2afcafdab55e789e578e23ef27365d035cb129dfada836"} Sep 11 14:53:47.901664 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.901616 2571 scope.go:117] "RemoveContainer" containerID="8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4" Sep 11 14:53:47.910762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.910744 2571 scope.go:117] "RemoveContainer" containerID="8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4" Sep 11 14:53:47.911066 ip-10-0-141-198 kubenswrapper[2571]: E0911 14:53:47.911047 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4\": container with ID starting with 8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4 not found: ID does not exist" containerID="8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4" Sep 11 14:53:47.911130 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.911076 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4"} err="failed to get container status \"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4\": rpc error: code = NotFound desc = could not find container \"8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4\": container with ID starting with 8319eacbadec7c81baf746fdaf46280fc5679932e95d638b8974c23a6460f5f4 not found: ID does not exist" Sep 11 14:53:47.927431 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.927407 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:53:47.932809 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.932788 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-rkdch"] Sep 11 14:53:47.964838 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:53:47.964804 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" path="/var/lib/kubelet/pods/f7be5fdc-0e9c-46c5-80c1-0dd9c279c989/volumes" Sep 11 14:54:07.231624 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:07.231595 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77989fd775-bkppn_9d590040-f0e5-4e43-9b46-7ebeaa6d15b4/manager/0.log" Sep 11 14:54:07.360499 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:07.360470 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bdsmq_bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4/server/0.log" Sep 11 14:54:07.904712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:07.904683 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-nkjm6_d6d57ada-91b2-4c4c-9711-1e80b10293ef/copy-manifests/0.log" Sep 11 14:54:07.913459 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:07.913432 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-nkjm6_d6d57ada-91b2-4c4c-9711-1e80b10293ef/manager/1.log" Sep 11 14:54:08.408684 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:08.408644 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b97d6fdf-ch47r_4175354e-fc94-415a-ba41-22273bb42031/maas-api/0.log" Sep 11 14:54:08.772500 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:08.772419 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bwsjb_cb5c02c2-9f96-4b6b-93ac-fa420330bfdb/postgres/0.log" Sep 11 14:54:10.935973 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:10.935921 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4gqt9_52f5f2dc-7b3d-4928-a72d-5b6df15c73e6/discovery/0.log" Sep 11 14:54:11.458358 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:11.458328 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-7xwq6_81aae6fa-1207-40cb-aef6-f7c32221ec45/payload-pre-processing/0.log" Sep 11 14:54:12.102714 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:12.102684 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-86ljs_e9f8a2a5-a5bd-4d2a-b147-09e8a314c315/main/0.log" Sep 11 14:54:12.478709 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:12.478635 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mxkqz_c38c1412-0037-4ce8-b791-56fb3a1209ee/main/0.log" Sep 11 14:54:12.722571 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:12.722538 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jr8sz_ea40923a-aea4-4d89-94f1-ab69c84e36f4/main/0.log" Sep 11 14:54:18.763515 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:18.763479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:54:18.836161 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:18.836131 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:54:21.888219 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:21.888187 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rv7pf_f74ee825-31fc-4842-8b9f-257b99bab445/global-pull-secret-syncer/0.log" Sep 11 14:54:21.948214 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:21.948181 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" Sep 11 14:54:22.107115 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:22.107084 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-198.ec2.internal_2471a11920facda24bd48e50635582ab/haproxy/0.log" Sep 11 14:54:28.782001 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:28.781971 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rpf9f_d611899b-7391-4314-873d-96b63b404f08/monitoring-plugin/0.log" Sep 11 14:54:28.935725 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:28.935698 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhl8v_2d3b1d86-eccb-45f7-9cb8-3285b8f04027/node-exporter/0.log" Sep 11 14:54:28.974553 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:28.974527 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhl8v_2d3b1d86-eccb-45f7-9cb8-3285b8f04027/kube-rbac-proxy/0.log" Sep 11 14:54:29.000729 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.000709 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhl8v_2d3b1d86-eccb-45f7-9cb8-3285b8f04027/init-textfile/0.log" Sep 11 14:54:29.290726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.290695 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/prometheus/0.log" Sep 11 14:54:29.322362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.322340 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/config-reloader/0.log" Sep 11 14:54:29.351521 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.351496 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/thanos-sidecar/0.log" Sep 11 14:54:29.379353 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.379331 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/kube-rbac-proxy-web/0.log" Sep 11 14:54:29.406708 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.406687 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/kube-rbac-proxy/0.log" Sep 11 14:54:29.436502 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.436480 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/kube-rbac-proxy-thanos/0.log" Sep 11 14:54:29.469248 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:29.469223 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b29786e6-3e12-4910-a2a6-80d7b4069a82/init-config-reloader/0.log" Sep 11 14:54:31.683309 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.683274 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl"] Sep 11 14:54:31.683770 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.683639 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" containerName="maas-api" Sep 11 14:54:31.683770 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.683650 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" containerName="maas-api" Sep 11 14:54:31.683770 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.683704 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7be5fdc-0e9c-46c5-80c1-0dd9c279c989" containerName="maas-api" Sep 11 14:54:31.686919 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.686896 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.691131 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.691111 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jc5ln\"/\"default-dockercfg-pzsf7\"" Sep 11 14:54:31.691762 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.691748 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"openshift-service-ca.crt\"" Sep 11 14:54:31.692157 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.692135 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"kube-root-ca.crt\"" Sep 11 14:54:31.700100 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.700078 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl"] Sep 11 14:54:31.773712 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.773683 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-sys\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.773861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.773732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrzv\" (UniqueName: \"kubernetes.io/projected/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-kube-api-access-xqrzv\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.773861 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.773796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-proc\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.773963 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.773864 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-lib-modules\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.773963 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.773894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-podres\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.874908 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.874876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrzv\" (UniqueName: \"kubernetes.io/projected/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-kube-api-access-xqrzv\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875057 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.874964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-proc\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875057 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875028 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-lib-modules\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875057 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875039 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-proc\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875057 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-podres\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-sys\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875155 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-podres\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875240 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875157 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-lib-modules\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.875370 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.875267 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-sys\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.884497 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.884474 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrzv\" (UniqueName: \"kubernetes.io/projected/c0fcea9a-3c32-469e-8ce6-3e55f54ab81c-kube-api-access-xqrzv\") pod \"perf-node-gather-daemonset-wxrgl\" (UID: \"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:31.891674 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.891651 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/2.log" Sep 11 14:54:31.896064 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.896045 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bc49w_fe0cc6a4-21f9-401c-8b7a-c76c4fb890db/console-operator/3.log" Sep 11 14:54:31.996663 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:31.996578 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:32.280320 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:32.280294 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl"] Sep 11 14:54:32.283370 ip-10-0-141-198 kubenswrapper[2571]: W0911 14:54:32.283342 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0fcea9a_3c32_469e_8ce6_3e55f54ab81c.slice/crio-b2e40fa78b86011ab747ad65354211b6132cb05c9d0a36610d9c6c4215c4df7a WatchSource:0}: Error finding container b2e40fa78b86011ab747ad65354211b6132cb05c9d0a36610d9c6c4215c4df7a: Status 404 returned error can't find the container with id b2e40fa78b86011ab747ad65354211b6132cb05c9d0a36610d9c6c4215c4df7a Sep 11 14:54:33.030409 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.030369 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7tmld_04ad6899-5f03-45f2-8c2e-9099ab141fa8/konnectivity-agent/0.log" limit=1024 Sep 11 14:54:33.107343 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.107309 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" event={"ID":"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c","Type":"ContainerStarted","Data":"5c8b5de6529ee358bf946e3acfeb1091f2597cee3c2c9c37ef0ccca804356c4b"} Sep 11 14:54:33.107546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.107383 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" event={"ID":"c0fcea9a-3c32-469e-8ce6-3e55f54ab81c","Type":"ContainerStarted","Data":"b2e40fa78b86011ab747ad65354211b6132cb05c9d0a36610d9c6c4215c4df7a"} Sep 11 14:54:33.107546 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.107407 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:33.138413 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.138358 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" podStartSLOduration=2.138343211 podStartE2EDuration="2.138343211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:33.136343651 +0000 UTC m=+2375.708001124" watchObservedRunningTime="2026-09-11 14:54:33.138343211 +0000 UTC m=+2375.710000685" Sep 11 14:54:33.376726 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:33.376648 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-wqd5v_b29c3578-ebff-44bf-b071-4819ce9f1a2e/volume-data-source-validator/0.log" Sep 11 14:54:34.879757 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:34.879722 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rbkk7_6a969f6c-8e60-463f-a6fe-c88428a90c63/dns/0.log" Sep 11 14:54:34.916567 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:34.916542 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rbkk7_6a969f6c-8e60-463f-a6fe-c88428a90c63/kube-rbac-proxy/0.log" Sep 11 14:54:35.003117 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:35.003085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5dl9d_be04675b-fc49-48cc-9121-3a49b2ecb6fb/dns-node-resolver/0.log" Sep 11 14:54:35.682605 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:35.682560 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f7f6bb8f9-khnkp_bb00318b-3c1b-49b2-8bd6-5f7403813874/registry/0.log" Sep 11 14:54:35.823306 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:35.823265 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n228s_b30ec984-eb09-4f9e-aa21-d12eaa0e73ad/node-ca/0.log" Sep 11 14:54:37.325685 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:37.325653 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4gqt9_52f5f2dc-7b3d-4928-a72d-5b6df15c73e6/discovery/0.log" Sep 11 14:54:37.489247 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:37.489210 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-7xwq6_81aae6fa-1207-40cb-aef6-f7c32221ec45/payload-pre-processing/0.log" Sep 11 14:54:38.538047 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:38.538015 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qv6jl_59ba9d55-7a02-453f-890b-28a0e451670c/serve-healthcheck-canary/0.log" Sep 11 14:54:39.108436 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.108404 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-g4f5j_f1380412-5d49-4c33-b973-cf9cabe5f316/insights-operator/0.log" Sep 11 14:54:39.109859 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.109841 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-g4f5j_f1380412-5d49-4c33-b973-cf9cabe5f316/insights-operator/1.log" Sep 11 14:54:39.122563 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.122538 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-wxrgl" Sep 11 14:54:39.226054 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.226026 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkghv_982c0ac4-150b-4d33-a391-9f70c3abaedb/kube-rbac-proxy/0.log" Sep 11 14:54:39.253852 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.253829 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkghv_982c0ac4-150b-4d33-a391-9f70c3abaedb/exporter/0.log" Sep 11 14:54:39.277043 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:39.277023 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkghv_982c0ac4-150b-4d33-a391-9f70c3abaedb/extractor/0.log" Sep 11 14:54:42.004698 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.004614 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77989fd775-bkppn_9d590040-f0e5-4e43-9b46-7ebeaa6d15b4/manager/0.log" Sep 11 14:54:42.043603 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.043581 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bdsmq_bdfc5a69-f7c0-40ea-bcf3-d024eb1304b4/server/0.log" Sep 11 14:54:42.319897 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.319854 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4gqt9_52f5f2dc-7b3d-4928-a72d-5b6df15c73e6/discovery/0.log" Sep 11 14:54:42.375808 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.375779 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-nkjm6_d6d57ada-91b2-4c4c-9711-1e80b10293ef/manager/1.log" Sep 11 14:54:42.394182 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.394153 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-nkjm6_d6d57ada-91b2-4c4c-9711-1e80b10293ef/manager/0.log" Sep 11 14:54:42.420467 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.420444 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-nkjm6_d6d57ada-91b2-4c4c-9711-1e80b10293ef/copy-manifests/0.log" Sep 11 14:54:42.432629 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:42.432606 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-7xwq6_81aae6fa-1207-40cb-aef6-f7c32221ec45/payload-pre-processing/0.log" Sep 11 14:54:43.756952 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:43.756913 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-744bf4ff48-n9zzq_2903db5b-3a78-49c2-a168-16515c8b5ee9/manager/0.log" Sep 11 14:54:44.435358 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:44.435266 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4gqt9_52f5f2dc-7b3d-4928-a72d-5b6df15c73e6/discovery/0.log" Sep 11 14:54:44.562362 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:44.562328 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-7xwq6_81aae6fa-1207-40cb-aef6-f7c32221ec45/payload-pre-processing/0.log" Sep 11 14:54:49.138185 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:49.138107 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-t8jfk_7c2fa79c-afc9-457a-9789-63db70c0a265/migrator/0.log" Sep 11 14:54:49.164918 ip-10-0-141-198 kubenswrapper[2571]: I0911 14:54:49.164888 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-t8jfk_7c2fa79c-afc9-457a-9789-63db70c0a265/graceful-termination/0.log"